Evaluation harness for MCP tool contracts: does a model pick the right tool, finish the task, and what does it cost.
Four tools with clear verb-noun naming (find_book, get_book, list_inventory, reserve_book). All have descriptions (194 - 250 chars, within baseline 34 - 392). Input schemas present with typed parameters and descriptions. v2 contract demonstrates awareness of quality gaps (v1 omits book_id from find_book results, lacks section/limit filters on list_inventory, returns generic 'Error' messages). However, output schemas are not formally documented in the code, inferred from descriptions and v1/v2 comparison. Error handling in v1 is deliberately poor (testing artifact); v2 improves it but still lacks structured error categories. No tool annotations (readOnlyHint/destructiveHint) visible. Composition is sound: tools chain naturally (find → get → reserve). Parameter constraints (section enum, limit bounds, ISO date format) are described but not enforced via JSON Schema enums/patterns.
Search the catalogue by words from a title or an author's name (case-insensitive). Returns up to 5 matches, each with book_id, title, author, section, price and stock. Use get_book with a book_id when you also need the shelf location.
Fetch one book by its book_id (format BK-0000, as returned by find_book or list_inventory). Returns title, author, section, price, stock and shelf.
List catalogue rows sorted by price ascending, optionally limited to one section (fiction, non-fiction, reference, children). Returns at most `limit` rows (default 20, max 50) plus the total count, so you can tell when rows were left out.
Reserve a book for in-store pickup. book_id is the BK-0000 id, customer_email is the customer's email address, and pickup_date must be an ISO date (YYYY-MM-DD), for example 2026-09-12. Returns a reservation_id, or a reason the reservation could not be made.
Output schemas not formally documented. Descriptions mention returned fields (book_id, title, author, section, price, stock, shelf) but no JSON Schema output type is visible in code. LLMs cannot reliably plan downstream calls without explicit output structure.
Parameter constraints (section enum: fiction|non-fiction|reference|children; limit: 1 - 50; ISO date format YYYY-MM-DD) are described in text but not enforced via JSON Schema enums or pattern fields. LLMs cannot parse text constraints reliably.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present. reserve_book is destructive (creates reservation) but not marked. find_book, get_book, list_inventory are read-only but unmarked. Agents cannot infer safety/retry semantics.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 70 | 2026-07-28+ | v2 |
Error handling in v1 returns generic 'Error' or 'Error: invalid request' with no recovery guidance. v2 improves this but code does not show structured error categories (retryable, user-fixable, fatal) or actionable messages like 'Invalid date format: expected YYYY-MM-DD'.
list_inventory accepts optional 'section' parameter but no enum constraint visible in schema. Description lists valid values (fiction, non-fiction, reference, children) as text; LLMs may hallucinate invalid sections like 'fiction-new' or 'sci-fi'.