Loading ToolBench
MCP server for FDA Orange Book API access with Code Mode execution, GraphQL/SPARQL support, and auto-staging of large responses
Server defines 5 tools with reasonable descriptions and schemas. All tools have descriptions (50-300 chars, within baseline 34-392 range). Input schemas present with type definitions and parameter descriptions. However, output schemas are not documented, LLMs cannot predict return structure. Error handling guidance is minimal. Tool names are action-oriented (search, execute, graphql_execute, sparql_execute) but lack verb_noun clarity for some (execute is generic). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all being READ_ONLY. Parameter descriptions are present but some lack format/constraint details (e.g., 'code' param in execute tools lacks guidance on what functions/APIs are available in the sandbox).
Search the API catalog (endpoints). Returns matching endpoints with full parameter docs. Use this to discover API capabilities before calling execute. Categories are listed. USAGE IN execute: api.get(path, params) for GET, api.post(path, body, params) for POST. Path params like /lookup/{id} are auto-interpolated from params. Remaining params become query string. Large responses (>100KB) are auto-staged: check result.__staged, return the staging info, use query_data to explore. Use limit/pagination params to control response size.
Execute JavaScript code against the API. Code runs in a sandboxed V8 isolate with: searchSpec(query) / searchPaths(query) — search the API spec; listCategories() / listTags() — inspect tags/categories; getEndpoint(path) / getOperation(idOrPath) — get endpoint docs; describeEndpoint(path, method?) / describeOperation(idOrPath) — format endpoint docs; spec — full frozen OpenAPI spec object; api.get(path, params) for GET, api.post(path, body, params) for POST; Path params like /lookup/{id} are auto-interpolated from params; Remaining params become query string; Large responses (>100KB) are auto-staged: check result.__staged, return the staging info, use query_data to explore; Use limit/pagination params to control response size. Large datasets auto-stage for SQL queries.
Execute JavaScript code against a GraphQL API. Code runs in a sandboxed V8 isolate with: gql.query(queryString, variables?) — GraphQL execution through host; schema.types(), schema.type(), schema.search() etc. — introspection helpers; db.stage(), db.queryStaged(), api.query() — staging helpers; console.log() capture. Optionally also a SECOND, REST upstream (api.get/api.post) routed through the same host proxy + DO-SQLite staging (hybrid GraphQL+REST).
Search the API spec. Write JavaScript code to search the OpenAPI spec, or use the legacy query/category arguments for keyword search. Available functions: searchPaths(query, maxResults=10) — keyword search across paths, summaries, tags, parameters; listTags() — list all tags with operation counts; getOperation(idOrPath) — get full operation by operationId or path; describeOperation(idOrPath) — formatted documentation for an operation; searchSpec/query helpers are also available for backward compatibility inside execute(); spec — the full frozen OpenAPI spec object (spec.paths, spec.info, etc.). Use search to discover endpoints, then write code in execute to call them.
Output schemas not documented. LLMs cannot predict return structure (fields, types, pagination). Baseline requires 100% of A+ tools to have documented return types.
Tool names lack verb_noun clarity. 'execute' and 'graphql_execute' are generic action verbs that do not convey what happens. Baseline: 90% of A+ tools start with specific verbs (get, create, search, update). Consider: 'execute_rest_api', 'execute_graphql_query', 'execute_sparql_query'.
No tool annotations despite all tools being READ_ONLY. Missing readOnlyHint in schema. Agents cannot distinguish safe tools from destructive ones without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | 2026-07-28+ | v2 |
Execute JavaScript code against a SPARQL endpoint. Code runs in a sandboxed V8 isolate with: sparql.query(queryString, opts?) — run SELECT/CONSTRUCT/DESCRIBE; returns array of plain { variable: value } rows; sparql.ask(queryString) — run ASK; returns boolean; sparql.raw(queryString, opts?) — return unparsed { head, results } envelope; prefixes — object of common ontology prefixes (rdfs, owl, xsd, skos, dcterms, void, obo, uberon, go, ncbigene, efo, obi, sio, up, ensembl, bgee); prefixHeader — pre-built `PREFIX …` block to prepend to your queries; console logging (log, warn, error, info) — captured output. Large SPARQL responses (>30KB) are auto-staged into SQLite.
'code' parameter in execute/graphql_execute/sparql_execute lacks format constraints. Description mentions available functions but does not specify: max code length, allowed imports, timeout, or error recovery. LLMs may pass invalid or unsafe code.
Error handling guidance missing. No recovery hints (e.g., 'If query fails, try catalog_search first'). Baseline pattern requires: 'Error responses must tell the LLM what to do next.'