Loading ToolBench
A code generation framework that transforms OpenAPI specifications into MCP (Model Context Protocol) servers using a domain-specific language (DSL). Generates TypeScript MCP tools with authentication, access control, and hook pipelines.
6 tools with visible schemas and descriptions, but significant quality gaps. All tools have input schemas with basic types (string, integer, object), but descriptions are verbose (200-400+ chars) and lack LLM-optimized clarity. Parameter descriptions exist but are often buried in lengthy prose mixing intent, API details, and implementation notes. No enum constraints on parameters. Output schemas are undocumented. Error handling descriptions mention HTTP codes but lack recovery guidance. Tool names follow verb_noun pattern (list*, get*) but descriptions conflate API documentation with agent-facing guidance.
Intent: Fetch balance for one account by accountId (from listMyAccounts). Users may only read their own accounts; admin may read any account. MCP arguments: pass accountId as top-level tool arguments. Do not nest path or query parameters under pathParams or query. Meta: operationId: get-account-balance Example: Balance for acc-alice-checking Response: HTTP 200 — accountId, balance, currency. Documented errors: HTTP 401 — Missing or invalid portal token HTTP 403 — Account not accessible HTTP 404 — Unknown account Runtime: protected — implement checkToolAccessForGetAccountBalance in src/hooks/api2ai/banking-tools/checkToolAccessForGetAccountBalance.ts; credential sent as header "Authorization" (prefix applied to the secret).
Intent: Admin only: list all demo accounts with balances. Meta: operationId: list-all-accounts Example: List all customer accounts as admin Response: HTTP 200 — role and accounts array. Documented errors: HTTP 401 — Missing or invalid portal token HTTP 403 — Admin role required Runtime: protected — implement checkToolAccessForListAllAccounts in src/hooks/api2ai/banking-tools/checkToolAccessForListAllAccounts.ts; credential sent as header "Authorization" (prefix applied to the secret).
Intent: Admin only: list bookings across all customers (Bearer JWT role=admin). Query limit caps how many bookings are returned (default 10, max 10). checkToolAccess + prepareToolCall demo — role gate before upstream call, limit in prepare stub. MCP arguments: pass limit as top-level tool arguments. Do not nest path or query parameters under pathParams or query. API: Requires Bearer JWT with role=admin. Returns bookings from all customers, capped by limit (max 10). Meta: operationId: list-all-bookings Example: List up to 10 bookings from all customers Response: HTTP 200 Cross-customer booking list content-type: application/json properties (top-level): bookings, limit, role Documented errors: HTTP 401 — Missing or invalid token HTTP 403 — Admin role required Runtime: protected — implement checkToolAccessForListAllBookings in src/hooks/api2ai/bookings-tools/checkToolAccessForListAllBookings.ts and prepareToolCallForListAllBookings in src/hooks/api2ai/bookings-tools/prepareToolCallForListAllBookings.ts; credential sent as header "Authorization" (prefix applied to the secret).
Descriptions conflate API documentation with agent guidance. Lengthy prose (250-400 chars) mixes intent, HTTP codes, JWT claims, and implementation hooks. LLMs cannot extract actionable intent from this noise.
No output schemas documented. Tools return HTTP 200 with JSON (e.g. customerId, accounts array, balance, currency) but LLMs cannot plan downstream calls or extract fields without knowing response structure.
Parameter descriptions lack constraints. 'limit' parameter in listVacationRentals and listAllBookings states 'default 10, max 10' in prose but no min/max schema constraints. LLMs cannot validate inputs.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | <=2025-11-25 | v2 |
Intent: List bookings for the authenticated customer (Bearer JWT). Path customerId is optional: when empty or omitted, filled from JWT claim customerId. Role user: path customerId must match JWT; role admin may list any customerId. Returns bookingId, unitId, checkIn, checkOut for each stay. MCP arguments: pass customerId as top-level tool arguments. Do not nest path or query parameters under pathParams or query. API: Requires Bearer JWT; role=user path customerId must match JWT claim; admin may read any customer. Meta: operationId: list-customer-bookings Example: List my bookings Response: HTTP 200 Booking list content-type: application/json properties (top-level): bookings, customerId Documented errors: HTTP 401 — Missing or invalid token HTTP 403 — Token customerId does not match path Runtime: protected — implement checkToolAccessForListBookings in src/hooks/api2ai/bookings-tools/checkToolAccessForListBookings.ts and prepareToolCallForListBookings in src/hooks/api2ai/bookings-tools/prepareToolCallForListBookings.ts; credential sent as header "Authorization" (prefix applied to the secret).
Intent: List bank accounts for the authenticated customer (portal JWT after token exchange). Requires banking:read permission in the portal token. Meta: operationId: list-my-accounts Example: List my accounts Response: HTTP 200 — customerId and accounts array (accountId, label, currency). Documented errors: HTTP 401 — Missing or invalid portal token Runtime: protected — implement checkToolAccessForListMyAccounts in src/hooks/api2ai/banking-tools/checkToolAccessForListMyAccounts.ts; credential sent as header "Authorization" (prefix applied to the secret).
Intent: List Ferienwohnungen (vacation rental units) — public, no login. Returns availability periods per unit (no guest names). Query limit caps how many units are returned (default 10, max 10). MCP arguments: pass limit as top-level tool arguments. Do not nest path or query parameters under pathParams or query. API: Public endpoint — no auth. Returns availability periods per unit (no guest identity). Use query limit to cap how many units are returned (max 10). Meta: operationId: list-vacation-rentals Example: Show up to 10 vacation rentals and their free/occupied periods Response: HTTP 200 Vacation rental units (public view) content-type: application/json properties (top-level): limit, units Runtime: implement prepareToolCallForListVacationRentals in src/hooks/api2ai/bookings-tools/prepareToolCallForListVacationRentals.ts (types from this tools module; run build:generated for .js).
Error handling lacks recovery guidance. Descriptions list HTTP codes (401, 403, 404) but do not tell LLM what to do next. E.g. 'HTTP 404, Unknown account' should suggest 'Call listMyAccounts() to find valid accountId values.'
Tool descriptions mention implementation details (checkToolAccessFor*, prepareToolCallFor* hooks, JWT claims, Bearer tokens) that are irrelevant to agent selection. This noise obscures the actual intent.