Loading ToolBench
An MCP server that dynamically loads tools from an OpenAPI specification.
The FastMCP OpenAPI Server dynamically registers tools from OpenAPI specifications. Only one logical tool exists: 'dynamic_openapi_operation', which is a meta-tool that wraps arbitrary OpenAPI operations. Critical issues: (1) The tool name is generic and does not follow verb_noun convention, it should reflect what OpenAPI operations actually do (e.g., 'invoke_openapi_operation' is marginally better, but the core problem is deeper). (2) Parameter schemas vary per operation and are not statically visible in the source code, they are derived at runtime from the OpenAPI spec. This violates the principle that schemas must be verifiable in source code. The 'Input' field states 'Schema varies per operation' with only a note that parameters are 'extracted from OpenAPI spec', but no actual JSON Schema is provided in the tool registration. (3) Tool description is minimal (67 chars) and does not explain WHEN to use this tool, what it returns, or how errors are handled. (4) The code in register_openapi_tools.py dynamically generates tool signatures using inspect.Parameter and functools.partial, but the actual tool registration call (mcp_server.add_tool) is not shown, we cannot verify that descriptions, input schemas, and output schemas are properly formatted for the MCP protocol. (5) Parameter descriptions are generated from the OpenAPI spec's parameter descriptions, which may be absent or poor quality in the upstream spec, no validation or enhancement is visible. (6) No output schema is documented; the code returns response.json() or response.text, but LLMs have no way to know the structure of returned data. (7) Error handling is minimal: a try/except that catches all exceptions and falls back to response.text is not actionable for LLMs. (8) The tool is not idempotent, calling the same OpenAPI operation twice may produce side effects (create duplicate records, charge twice, etc.), but there is no warning or confirmation mechanism.
Dynamically registered tools from OpenAPI specification. Each OpenAPI operation becomes a tool with parameters derived from the spec's path, query, header, and request body parameters.
Tool name 'dynamic_openapi_operation' is generic and does not follow verb_noun convention (pattern:tool). Names like this force LLMs to read the description to understand what the tool does, which wastes reasoning cycles. The name should reflect the action (invoke, execute, call) and the resource (openapi_operation, api_endpoint).
No input schema is visible in the source code. The tool registration code dynamically generates parameter signatures from the OpenAPI spec at runtime, but the actual schema passed to mcp_server.add_tool is not shown. Per HARD SCORING RULES, if a tool has NO input schema at all, schema score MUST be 0. The 'Input' field states schemas vary per operation with only a descriptive note, this is not a verifiable schema and does not meet the requirement that 'Schema format: input schemas use proper JSON Schema with types and descriptions'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 28 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Tool description is too short (67 chars) and lacks critical context. Per pattern:tool-description, descriptions must be 10 - 1024 chars, but more importantly, they must answer: What does it do? When should the LLM call it instead of a similar tool? What does it return? The current description ('Dynamically registered tools from OpenAPI specification. Each OpenAPI operation becomes a tool...') does not explain when to invoke this tool, what OpenAPI operations are available, or what the expected response structure is.
No output schema is documented. The code returns response.json() or response.text depending on whether parsing succeeds, but LLMs have no way to know what fields to expect in the response, what type each field is, or how to chain this tool's output to downstream tools. Per pattern:tool, 'Document the output schema. LLMs need to know what fields to expect so they can plan downstream tool calls and extract the right data.'
Error handling does not guide recovery. The try/except block catches all exceptions and falls back to response.text. This gives LLMs no actionable information: was the API unreachable? Was the request malformed? Is the error retryable? Per pattern:recovery-guide, 'Error responses must tell the LLM what to do next.' A raw error message or text dump is not actionable.
No idempotency or confirmation mechanism for destructive operations. The tool wraps arbitrary OpenAPI operations, including DELETE, PUT, and POST calls that modify state. There is no warning in the description that this tool can delete data, create duplicate records, or trigger irreversible side effects. Per pattern:confirmation-request, 'Irreversible operations (delete, send, publish) should support a dry-run or confirmation step. Agents make mistakes, a confirm_before_execute pattern prevents catastrophic errors.'
Parameter descriptions may be absent or poor quality. The code extracts param descriptions from the OpenAPI spec, but there is no validation or enhancement. If the upstream OpenAPI spec lacks descriptions (common in legacy APIs), the tool will not meet the requirement that 'Every parameter needs a description.' The generated parameter list is not sanitized against vague or example-laden descriptions.
No secrets handling or credential management visible. The code loads an OpenAPI spec and invokes operations via httpx.get() and client.invoke_operation(), but there is no evidence of secure credential injection. If the OpenAPI spec requires auth headers (OAuth2 tokens, API keys), these must not appear as tool parameters. The __main__.py accepts oauth2_client_id and oauth2_client_secret as command-line args, but it is unclear whether these are stored securely or passed to the client.