Loading ToolBench
A multi-server MCP demonstration for an airline booking scenario with three purpose-built servers: reservations (flight search and booking), loyalty (frequent-flyer miles lookup), and policies (fare rule search). Designed to showcase tool reliability, data freshness, and error handling across multiple beats.
Four tools with complete input schemas and descriptions. Naming is clear and action-oriented (search_*, book_*, check_*). Descriptions are detailed (100-250 chars) and explain purpose, return shape, and constraints. However, output schemas are documented only in prose within descriptions, not as formal JSON Schema. Parameters lack type constraints (enums, min/max). Error handling is mentioned but not formalized in tool definitions. No tool annotations (readOnlyHint/destructiveHint). The loyalty server demonstrates good fail-closed error semantics, but this is not reflected in schema-level error documentation.
Book one flight. SCOPED WRITE: this inserts exactly ONE row into the `bookings` table (a fresh BKG-60xx id, the given flight_id and passenger, miles_to_apply, status 'confirmed') and does nothing else. It does not expose arbitrary SQL and cannot read or modify anything but this one insert. `seat` is accepted for interface completeness but not persisted (the demo bookings schema has no seat column). Returns: {booking_id, flight_id, passenger, miles_applied, status}.
Look up a frequent-flyer member's miles balance and tier. READ ONLY. Returns: {name, miles_balance, tier}. Fails closed on purpose: if the loyalty database is missing, corrupt, or unreadable, or the member is not found, this RAISES instead of returning a value. The host sees a failed tool call -- it must never mistake "could not check" for "the miles are fine."
Search this weekend's flights leaving `origin` (default SJC), each row carrying the DESTINATION's current weather from the reservations backend. Returns a list of dicts: {flight_id, airline, origin, destination, city, date, price_usd, seats_left, weather, weather_as_of} `city`, `weather`, and `weather_as_of` come from a LEFT JOIN of airports on destination = code, so every flight reports its destination's weather and how fresh that reading is (weather_as_of). Pass `date` (YYYY-MM-DD) to filter to a single day; omit it to see the whole weekend.
Keyword-search the active fare-rule / policy markdown files and return the best-matching passage, quoted verbatim, with its source file's version and freshness. Returns: {passage, source_file, version, last_updated}. If nothing matches, `passage` explains that and `source_file` is null. Reads policy/active/*.md fresh on every call, so swapping the active policy (current <-> superseded) is reflected on the next search.
Output schemas documented only in prose descriptions, not as formal JSON Schema. LLMs cannot parse unstructured text to infer response structure for downstream tool chaining.
Input parameters lack type constraints (enums, min/max, patterns). 'origin' accepts any string; 'date' has no format validation; 'miles_to_apply' has no bounds. LLMs will pass invalid values.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Agents cannot distinguish safe reads from destructive writes without parsing descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
Error handling is implicit in code (check_miles raises on missing DB, book_flight has no documented error cases). Tool definitions do not declare error conditions or recovery guidance.
search_flights and search_policies return unbounded lists. No pagination, limit, or total_count documented. Large result sets will blow context windows.