Loading ToolBench
FastMCP is a fast, Pythonic framework for building MCP servers and clients. Repository contains FastMCP core library plus multiple example MCP servers including Bacularis, Bconsole, Proxmox, weather, and others.
This evaluation covers the FastMCP framework example servers. The server collection includes 6 tools across multiple example files. Overall quality is below production baseline: tool descriptions vary in completeness, several parameter schemas lack descriptions, and definitions show inconsistent polish. Tools like 'ask_assistant' and 'call_tools_bulk' have good structure, but others like 'get_example_data' and 'get_json_data' are trivial examples with minimal parameter validation. The bulk tool callers ('call_tools_bulk', 'call_tools_bulk') are well-defined with structured input schemas, but lack comprehensive error guidance. None of the tools declare risk/security models in descriptions (only via external metadata), and no tool annotations (readOnlyHint, destructiveHint) are visible in the code.
Ask the assistant a question. It can use tools to help answer.
Call a single tool registered on this MCP server multiple times with a single request. Each call can include different arguments. Useful for speeding up what would otherwise take several individual tool calls.
Call multiple tools registered on this MCP server in a single request. Each call can be for a different tool and can include different arguments. Useful for speeding up what would otherwise take several individual tool calls.
Echo text back with metadata about the operation.
Returns some example data serialized as YAML.
Returns data with default JSON serialization.
get_example_data and get_json_data have trivial descriptions ('Returns some example data serialized as YAML.', 'Returns data with default JSON serialization.') that do not explain purpose, when to use, or output structure. Descriptions should be 50-200 chars and answer: what does it do, when should the LLM call it, what does it return?
No tool annotations visible in code. Tools lack readOnlyHint, destructiveHint, or idempotentHint declarations. The 'call_tools_bulk' and 'call_tool_bulk' tools are marked WRITE risk externally, but the tool definitions do not include annotations that would inform the MCP client about their destructive nature.
Parameter 'tool_arguments' in 'call_tool_bulk' has an items schema with only 'type' declared as 'string|integer|number|boolean|null' but no inner properties or description. The schema is malformed: items should define an object with 'properties', not a union type. This makes validation impossible and the parameter essentially untyped.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 53 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
ask_assistant tool calls back into the sampling capability (deprecated as of spec 2026-07-28), which requires server-initiated calls to the client. This is a deprecated pattern that should be replaced by direct LLM provider integration.
No error handling guidance in tool descriptions. Tools like 'call_tools_bulk' and 'call_tool_bulk' accept 'continue_on_error' but do not document what happens when errors occur: are failures returned per-item, in aggregate, or as exceptions? LLMs cannot plan recovery without this clarity.
Output schemas are not documented for any tool. Descriptions do not specify what fields the response will contain, what types they are, or how to chain them into subsequent tool calls. This forces LLMs to infer output structure on the fly.
echo tool description does not explain what 'metadata about the operation' means. Is this execution time, log level, call depth, timestamp? Without specificity, LLMs cannot predict or use the output.