Loading ToolBench
Local integration model demonstrating MCP tool dispatch, reservation systems, and asset inspection workflows with two candidate implementations (A and B) for comparison
This is a well-structured engineering review lab with 6 tools demonstrating solid patterns. Tools have clear verb-noun naming (catalogue_lookup, reserve_samples, find_assets, read_inspection, record_inspection). All visible tools include descriptions (194 - 250 chars, within baseline 34 - 392 range) and complete JSON schemas with type constraints. Input validation is implemented. However, tool annotations (readOnlyHint/destructiveHint) are declared in FEATURES but not visible in the actual schema definitions in the code. Output schemas are not documented, responses use structuredContent but the shape is not formally specified. Error handling provides actionable messages ('Unknown SKU', 'Insufficient stock') but lacks recovery guidance. The server demonstrates idempotent write patterns (request_key for deduplication) and proper composition (find_assets → read_inspection → record_inspection chain). Two tools (find_assets, read_inspection, record_inspection) are in a different file (evaluations/2026-09-12/build-output/demo.py) and their full definitions are not shown, so those are capped at 50 per the inference rule.
Look up one sample item, its sealed pack size, and available stock.
Find equipment by case-insensitive name substring, optionally at an exact site ID. Only assigned sites are searched. Results include IDs and site names to distinguish similar assets; use read_inspection on the chosen asset. Empty query lists all visible equipment.
Read the latest inspection and current asset revision, or a specific saved inspection using receipt.inspection.id. asset.revision is current; inspection.revision identifies that inspection's commit. Seed data has unknown recorded_by and recorded_at (null).
Append an inspection for an operator's assigned site, guarded by the asset revision from a fresh latest read. On stale revision, read latest and reconsider before retrying. Choose a request_id once per intended write. If a reply is lost, repeat ALL arguments unchanged: the original receipt is returned even after later writes. A saved key reused with any changed argument is rejected. Keys are scoped to principal and this operation, retained only for this in-memory notebook instance. Rejected writes do not reserve keys.
Recover the receipt for a logical reservation using its request key.
Tool annotations declared in FEATURES but not visible in schema definitions. toolAnnotations=true but readOnlyHint/destructiveHint/idempotentHint not present in inputSchema objects.
Output schemas not documented. Responses use structuredContent but the shape (fields, types, required properties) is not formally specified. LLMs cannot plan downstream calls without knowing response structure.
Error messages lack recovery guidance. 'Unknown SKU' and 'Insufficient stock' are clear but do not suggest next steps (e.g., 'Try catalogue_lookup() to see available SKUs').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2025-06-18+ | v2 |
Create a local reservation for catalogue stock and return a receipt.
Three tools (find_assets, read_inspection, record_inspection) defined in separate file not fully visible in source. Per inference rule, capped at 50 overall.
Parameter 'query' in find_assets has minLength=0, allowing empty string. Description says 'Empty query lists all visible equipment' but this is not enforced by schema, LLMs may pass null or omit it.