Loading ToolBench
MCP gateway with schema deferral + response shielding. Replaces 12+ MCP servers with 6 tools, saves ~99% context tokens and ~2.7GB RAM via shared HTTP daemon mode.
Strong schema definitions with Zod validation and clear descriptions. All 7 tools have explicit names, descriptions, and typed input schemas. Tool names follow verb_noun pattern (search, describe, invoke, get_result, status). Descriptions are detailed (100-250 chars) and explain WHAT, WHEN, and HOW. However, output schemas are not documented, LLMs cannot predict response structure. Error handling is minimal (no recovery guidance, no categorization). Parameters lack constraints (e.g., limit has no min/max bounds despite mentioning 'max 50'). No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels.
Get full details for a specific tool including its complete input schema. Use the tool ID from gateway.search results (format: serverKey::toolName). Most tools return fieldNames in search results — describe is only needed for full schema detail.
Retrieve the full result of a truncated tool response. Use the _ref value from a truncated gateway.invoke response. Supports pagination (offset/limit), field projection, and text search. For arrays: offset and limit paginate through items. For strings: offset is character position. Use 'fields' to project specific keys from array items (reduces token usage). Use 'search' to filter items containing specific text.
Execute a tool on an upstream MCP server synchronously. Response is automatically truncated if large — check for _ref field in the result. If _ref is present, use gateway.get_result to paginate through the full response. Always call gateway.describe first to know the correct argument format.
Start an asynchronous tool execution. Returns a job ID immediately. Use gateway.invoke_status to poll for completion. Useful for long-running tools (web search, E2E tests, etc.).
Check the status of an async job. Returns status, result (if completed), or error (if failed).
Output schemas not documented. LLMs cannot predict response structure for any tool. gateway.invoke returns truncated responses with _ref field, but the full response schema is not specified. gateway.search returns 'tool IDs, names, displayNames, fieldNames, descriptions, relevance scores' but no formal schema.
Numeric parameters lack constraints. 'limit' parameter described as 'max 50' but no min/max bounds in schema. 'offset' and 'priority' similarly unconstrained. Zod schema uses z.number().optional() without .min()/.max() guards, allowing LLMs to pass invalid values.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2025-06-18+ | v2 |
Search for tools across all connected MCP servers using BM25 scoring with fuzzy matching. Returns tool IDs, names, displayNames, fieldNames, descriptions, and relevance scores — NOT full schemas. An empty query returns all available tools.
Returns the current status of the gateway including connected servers, tool counts, config file path, last reload timestamp, pending reload status, and available codegraph projects. Use this to check if config reloaded after changes.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint). gateway.invoke and gateway.invoke_async are marked WRITE risk but lack destructiveHint annotation. gateway.search, gateway.describe, gateway.invoke_status, gateway.get_result, gateway.status are READ_ONLY but lack readOnlyHint. Agents cannot infer safety/idempotency from tool definitions.
Error handling lacks recovery guidance. No error categorization (retryable vs user-fixable vs fatal). gateway.invoke description mentions 'check for _ref field' but does not explain what happens on timeout, network failure, or upstream tool error. No guidance on what the LLM should do next.
gateway.status description is vague ('Returns the current status of the gateway'). Does not explain what 'status' means or when to call it. No mention of what fields are returned or how to interpret 'pending reload status'.