Loading ToolBench
FastMCP server exposing web/news/RAG/time-weather-stocks tools
This server exposes 21 tools across web search, RAG, time/weather/stock domains. The tool definitions are visible in server.py with explicit @app.tool() registrations. All tools have non-empty descriptions and input schemas. However, there are significant gaps in parameter-level descriptions, output schema documentation, and error handling guidance. Many descriptions are generic or lack contextual depth about when to use each tool. Parameter descriptions are often minimal (e.g., 'Place name' without format guidance). Output schemas are implicit, the docstrings describe return types with dict[str, object] but do not document the expected fields or structure. Error handling returns generic error messages (e.g., str(e)) without recovery guidance. The tool naming follows verb_noun patterns (web_search_tool, fetch_url_tool, docs_add_document) which is good, but some names are slightly verbose (_tool suffix is redundant). Tool composition is reasonable, each tool does one thing, but tools that operate on the same resource (e.g., docs_add_document, docs_add_directory, docs_remove, docs_search, docs_list, rag_answer_tool, rag_status_tool, rag_rebuild, rag_add_url, rag_add_crawl) could benefit from clearer naming distinctions. Some tools accept user_id and chat_id as required parameters without documenting what these mean or whether they must be pre-existing identifiers. Output includes 'sources' field for citation uniformity (web_search_tool, news_search_tool, fetch_url_tool, fetch_url_readable_tool, crawl_site_tool), which is a good pattern, but other tools (stock_quote_tool, weather_now_tool) do not. Pagination is absent, tools like docs_list, docs_search do not accept limit/offset parameters despite potentially returning many results.
Perform a small crawl rooted at the given URL.
Add all documents in a directory to the user's RAG index.
Add a document to the user's RAG index.
List all documents indexed for a user.
Remove a document from the user's RAG index.
Search documents in the user's RAG index.
Fetch a URL and extract readability-style main content.
Missing or minimal parameter-level descriptions. Parameters like 'place', 'tz', 'max_results', 'user_id', 'chat_id' lack guidance on format, valid values, or constraints. LLMs cannot infer whether 'place' should be 'New York', 'NYC', or 'US/Eastern', or whether user_id is a UUID, email, or opaque identifier.
Output schemas are not documented. Tool descriptions describe return types as dict[str, object] but do not specify which fields are expected, their types, or when they will be present. LLMs cannot plan downstream actions without knowing what fields they will receive. Example: docs_list returns DocumentListResult, but the caller cannot see the structure without inspecting the schema definition elsewhere.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 53 | - | v1 |
Fetch a URL and return a lightweight representation.
Search for news articles using DuckDuckGo News.
Crawl a website and add all pages to the user's RAG index.
Add content from a URL to the user's RAG index.
Generate an answer to a question using RAG (Retrieval-Augmented Generation).
Rebuild the RAG index for a user.
Get RAG system status for a user.
Get stock quote for a specified symbol.
Get stock quotes for multiple symbols.
Get the current local time in a specified timezone.
Get the current local time in a specified place.
Get daily weather forecast for a specified place.
Get current weather for a specified place.
Search the web and return the top N results. Always returns unified ``sources`` list so downstream agents can extract citations uniformly.
Error handling returns generic exception strings (str(e)) without recovery guidance. Example: web_search_tool and fetch_url_tool catch all exceptions and return error: str(e). An LLM seeing 'error: [Errno -2] Name or service not known' has no guidance on whether to retry, ask the user, or abort. No categorization of errors as retryable, user-fixable, or fatal.
No pagination support. Tools like docs_list and docs_search do not accept limit/offset or page parameters and do not return result counts. If a user has 1000 indexed documents, docs_list returns all of them, wasting tokens and potentially hitting context limits. Similarly, docs_search does accept a 'k' param but returns all matching results without pagination.
Tool names contain redundant '_tool' suffix (e.g., web_search_tool, news_search_tool, weather_now_tool, stock_quote_tool). The suffix is unnecessary, the tool is registered in the tools list and the suffix adds no semantic value. Standard naming is 'web_search', 'news_search', etc. This is minor but affects code clarity.
Inconsistent use of 'sources' field. web_search_tool, news_search_tool, fetch_url_tool, fetch_url_readable_tool, and crawl_site_tool all return a 'sources' list for citation tracking (good pattern). However, time_local, weather_now_tool, stock_quote_tool do not. This inconsistency means agents cannot extract citations uniformly across different tool categories, they must handle some tools as having sources and others as not.
Parameter name ambiguity in RAG tools. Many RAG tools require 'user_id' and 'chat_id' but do not document what these are, whether they are UUIDs, strings, or references to a data model, and whether an agent must create them or they come from somewhere else. This forces agents to guess or experiment.
No destructive operation confirmation or dry-run. docs_remove is marked as DESTRUCTIVE but has no confirmation mechanism. An agent could accidentally delete all indexed documents. For destructive tools, the rubric recommends a dry-run or confirmation step pattern (e.g., confirm_delete or a force=false default).
Generic tool descriptions lack context about when to call each tool. Example: 'Search for news articles using DuckDuckGo News' does not say when to prefer it over web_search_tool (recency? news source authority? structured article metadata?). Without this guidance, LLMs guess or invoke both tools redundantly.