Loading ToolBench
A hand-written MCP server at the protocol level, four agent topologies over it, and the failure rate of each measured under a controlled fault model.
Five tools with explicit schemas, descriptions, and thoughtful error handling. Tool names are action-oriented (search_, fetch_, summarise_, append_, delete_). Descriptions are substantive (100-200 chars) and explain recovery paths. Schemas are complete with required fields. Idempotency and destructive hints are properly annotated. Key gaps: parameter descriptions are minimal or absent (e.g., 'record_ids' array lacks item type or count guidance); output schemas are not documented; no pagination support for list-like operations; error messages are described in docstrings but not validated in visible code.
Append a note to the server's note log. Non-idempotent: retries after timeout will double-write. Requires idempotency key for safe replay.
Delete a record. Destructive and gated: first call without confirm_token returns a token; second call must include that token and confirm=True. Prevents accidental deletion.
Fetch a single record by ID. Returns the full record object or error naming how to find valid IDs.
Search records by query string matching title, owner, or tags. Returns matching record IDs or error with recovery guidance.
Sum the amounts of specified records. Returns total or error if any record ID is unknown.
Parameter descriptions are sparse or missing. 'record_ids' in summarise_amounts lacks type detail (array of what? min/max length?). 'query' in search_records lacks format guidance (regex? case-sensitive? field priority?).
Output schemas are not documented. Tools return ToolResult with 'text' and 'isError', but the structure of 'text' content (JSON, plaintext, structured fields?) is not specified. Agents cannot plan downstream calls without knowing response shape.
search_records and fetch_record have overlapping descriptions ('record', 'matching'). The docstring acknowledges this is intentional for testing, but in production this violates the name-clarity pattern and will cause LLM confusion.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
append_note is non-idempotent but no idempotency_key parameter is visible in the input schema. The docstring states the server enforces deduplication via idempotency_key, but the schema does not expose this parameter for the agent to provide.
No pagination support. summarise_amounts accepts an unbounded array of record_ids. Large arrays could exceed context limits or cause performance issues. No limit guidance or batch variant offered.