Loading ToolBench
A demonstration MCP server with multiple tools including text manipulation, web search, Wikipedia lookup, and Python code execution. Provides both FastAPI (classic_api) and FastMCP implementations.
The MCP server has 9 tools with basic schemas and minimal descriptions. All tools have explicit registration via the @mcp.tool decorator in mcp_server_app.py, and all have type-annotated parameters. However, descriptions are uniformly sparse (10-30 characters), parameter descriptions are either absent or trivial, output schemas are not documented, and there is no evidence of error handling guidance or recovery paths. The tool suite mixes trivial utilities (add, reverse_string, greet) with high-risk operations (python_exec) without commensurate safety guardrails. No tools follow the verb_noun naming convention strongly (e.g., 'add_tool' should be 'add_numbers', 'web_search_tool' should be 'search_web'). No parameter constraints, enums, or validation rules are documented. The codebase shows tool implementations in separate files (tools/add.py, etc.) but those files are not provided, so we cannot verify output structure or error handling.
Add two numbers.
Greet someone.
Execute Python code.
Reverse a string.
Tool 1 implementation.
Tool 2 implementation.
Search the web.
Get Wikipedia summary.
Descriptions critically underdeveloped (10-30 characters). All 9 tools fail the 10-1024 character rubric baseline. Descriptions like 'Add two numbers.' and 'Reverse a string.' lack context for LLM tool selection, dependency hints, or state-mutation warnings.
Input parameters lack descriptions. All 9 tools have typed parameters but NO per-parameter documentation. 'a', 'b', 's', 'query', 'code', 'name', 'param' are registered with types (int, str) but zero guidance on format, constraints, or valid ranges. Rubric baseline: 100% of A+ tools have parameter descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 43 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 36 | - | v1 |
Count words in text.
Output schemas not documented. No tool declares its return type or structure. The server uses return type hints in Python (e.g., 'async def add_tool(a: int, b: int) -> int'), but the MCP protocol requires explicit output schema in tool definitions for LLM reasoning. LLMs cannot plan downstream calls without knowing what fields a tool returns.
Naming convention violations. 'add_tool', 'reverse_string_tool', 'word_count_tool', 'web_search_tool', 'python_exec_tool' all have '_tool' suffix, which is redundant in the MCP context (all are tools). Better names: 'add_numbers', 'reverse_string', 'count_words', 'search_web', 'execute_python'. Additionally, 'tool1_tool' and 'tool2_tool' are generic placeholders, the actual intent is unknowable.
Unguarded irreversible operation: 'python_exec_tool' has Risk=IRREVERSIBLE but zero confirmation, dry-run, or scope constraints. The description 'Execute Python code.' does not warn the agent that this operation can delete files, spawn processes, or exfiltrate data. No error handling or recovery guidance.
No error handling or recovery guidance. No tool description mentions what errors could occur, what they mean, or what the agent should do next. Rubric: 'Error responses must tell the LLM what to do next.' Agents have no fallback if a Wikipedia query fails or a web search times out.
No input validation or constraint documentation. 'add_tool' takes two integers with no min/max. 'word_count_tool' accepts any string with no length limit. 'python_exec_tool' accepts arbitrary code with no sandboxing hint. Rubric: 'Specify minimum and maximum for numeric parameters' and 'When a parameter has regex or length limit, state it in the description.'
Generic/placeholder tool names. 'tool1_tool' and 'tool2_tool' reveal nothing about intent. The code imports 'from tools.tool1 import run as tool1' but the tool1.py implementation shown only has 'question' → 'answer' mapping with no semantic meaning. These should be renamed to describe actual functionality (e.g., 'classify_text', 'summarize_document').
External API tools (wikipedia_tool, web_search_tool) lack pagination, result limits, and field stripping. No indication of how many results are returned, whether responses are paginated, or if metadata is stripped. Rubric: 'Tools returning lists should accept page/offset and limit parameters and return a total count.' Returning 500 Wikipedia hits or web search results blows the context window.
State-mutation warnings missing. 'web_search_tool' and 'wikipedia_tool' are read-only, but 'python_exec_tool' mutates system state. Descriptions do not clarify which tools are safe to retry and which are not. Rubric: 'If the tool modifies state (creates, updates, deletes, sends), the description must say so.'