Loading ToolBench
Monorepo for converting OpenAPI specifications into AI SDK tools and MCP servers. Includes stdio-mcp server that exposes OpenAPI endpoints as MCP tools via stdio transport.
The MCP server exposes 3 tools with significant quality gaps. Tool naming lacks action verbs and clarity. Descriptions are present but generic and fall short of LLM-optimized guidance. Parameter schemas for the primary tool (dynamic_openapi_tools) are not statically visible, they are generated at runtime from OpenAPI specs, making validation impossible. The 'search' and 'execute' tools have minimal parameter descriptions. Output schemas are not documented. Error handling and recovery guidance are absent. This is a mediocre implementation that would not pass production code review.
Dynamically generated tools from OpenAPI specification operations. Each operation in the spec becomes an executable tool with parameter validation, authentication handling, and HTTP request execution.
Execute tool for code mode - executes an OpenAPI operation by name with provided parameters
Search tool for code mode - searches through available OpenAPI operations to find matching endpoints for a user query
Primary tool 'dynamic_openapi_tools' has no statically defined schema. Parameter validation is deferred to runtime based on OpenAPI spec parsing. This violates the critical requirement that tools declare their input schemas upfront so LLMs can understand what parameters they accept before execution.
Tool names do not follow verb_noun convention. 'dynamic_openapi_tools' is a noun phrase describing what the tool is, not an action verb describing what it does (e.g., 'execute_openapi_operation', 'call_api'). 'search' and 'execute' lack the resource they operate on (e.g., 'search_operations', 'execute_operation'). LLMs cannot infer intent from these names.
Parameter descriptions are absent or trivial. The 'search' tool's 'query' parameter has a 2-word description; the 'execute' tool's 'params' parameter description is a single phrase. Per the rubric baseline, parameter descriptions average 72 chars for A+ tools. These fall far short and provide no guidance on what values are valid, required format, or constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 35 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 26 | - | v1 |
Tool descriptions are generic and lack LLM-optimized context. The descriptions state WHAT the tools do but not WHEN to use them, WHAT they return, or HOW they fail. E.g., 'dynamic_openapi_tools' says it's 'dynamically generated' but does not explain when an LLM should invoke it or what the result looks like.
No output schemas are documented. LLMs cannot plan downstream tool calls or extract expected fields if they do not know what structure the tools return. This violates the requirement that tools document return types.
Error handling is not visible in tool definitions. There is no guidance on what errors these tools can raise, whether they are retryable, or what recovery action the LLM should take. E.g., if 'execute' fails due to authentication, rate limiting, or invalid parameters, the LLM has no recovery path documented.
The 'execute' tool accepts an untyped 'params' object. No constraint is specified on what keys/values are valid, what format they should take, or whether they are required. LLMs will hallucinate arbitrary keys, causing execution failures.