Loading ToolBench
MCP server providing Code Mode execution tools for API discovery and querying, with support for REST APIs, GraphQL, and SPARQL endpoints
Critical gaps across all dimensions. Tool descriptions are present but lack actionable detail for LLM selection. Input schemas are visible but incomplete: parameters lack type definitions (e.g., 'code' in execute/graphql_execute/sparql_execute has no explicit type constraint). No output schemas documented. Error handling guidance absent. The 'execute' family of tools accept arbitrary JavaScript code, a severe security and composition risk without sandboxing guarantees, input validation, or recovery guidance. Tool names are verb-based (positive), but descriptions do not explain WHEN to use each tool vs. alternatives (catalog_search vs. search distinction unclear). No parameter descriptions for 'workspace' across three tools. Baseline: 549 production tools average 194-char descriptions with 100% param coverage; this server has ~100-char descriptions with ~60% param coverage.
Search the API catalog (endpoints). Returns matching endpoints with full parameter docs. Use this to discover API capabilities before calling execute. Categories: (dynamic based on catalog) USAGE IN execute: - api.get(path, params) for GET, api.post(path, body, params) for POST - Path params like /lookup/{id} are auto-interpolated from params: api.get('/lookup/{id}', {id: 'ENSG...'}) - Remaining params become query string - Large responses (>100KB) are auto-staged: check result.__staged, return the staging info, use query_data to explore - Use limit/pagination params to control response size. Large datasets auto-stage for SQL queries.
Execute JavaScript code against the API with access to api.get/api.post helpers, catalog search functions, and auto-staging for large responses. Code runs in a sandboxed V8 isolate.
Execute JavaScript code against a GraphQL endpoint with access to gql.query() for GraphQL execution, schema introspection helpers (schema.types(), schema.search(), etc.), and optional REST API support. Code runs in a sandboxed V8 isolate.
Search the API specification (OpenAPI mode). Write JavaScript code to search the OpenAPI spec, or use legacy query/category arguments for keyword search. Available functions: searchPaths(query, maxResults=10), listTags(), getOperation(idOrPath), describeOperation(idOrPath), spec object access.
Execute JavaScript code against a SPARQL endpoint with access to sparql.query(), sparql.ask(), sparql.raw() for SPARQL execution, common ontology prefixes, and auto-staging for large responses. Code runs in a sandboxed V8 isolate.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., what does execute() return? Is it structured JSON or free text?). Baseline: 100% of A+ tools document return types.
execute/graphql_execute/sparql_execute accept arbitrary code strings with no input validation, sandboxing verification, or error recovery guidance. Descriptions claim 'sandboxed V8 isolate' but provide no evidence of constraint enforcement, timeout handling, or injection prevention. This violates pattern:tool-gateway (treat agent input as untrusted).
'workspace' parameter in execute/graphql_execute lacks description. Baseline: 100% of A+ tools have param descriptions. LLMs cannot infer whether workspace is optional, what it controls, or when to use it.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 34 | 2026-07-28+ | v2 |
catalog_search and search both perform keyword search over API specs. Distinction unclear: when should LLM choose catalog_search vs. search? Descriptions do not explain the difference or guide selection. Violates pattern:tool (each tool should do exactly one thing with clear naming).
No error handling guidance. Descriptions do not explain what happens on failure (e.g., invalid code in execute, malformed SPARQL query, API timeout). Baseline: pattern:recovery-guide requires error responses to tell LLM what to do next.