Loading ToolBench
A banking support agent that uses MCP tools to lookup SOP policies, create escalation cases, and maintain audit trails for banking customer interactions.
Three tools with complete JSON schemas and basic descriptions, but significant gaps in parameter documentation and error handling. All tools have descriptions (10-50 chars), but lack depth for LLM decision-making. Parameters are typed but mostly undescribed. No output schemas documented. No error recovery guidance. Naming is clear (verb_noun pattern) but descriptions are minimal. This is a typical community server with functional definitions but insufficient LLM optimization.
Append an audit event for traceability.
Create an escalation case in the database.
Fetch SOP policy by intent_name.
Parameter descriptions missing or minimal. 'trace_id', 'intent_name', 'priority', 'event_type', 'payload' lack actionable guidance on format, constraints, or valid values. LLMs cannot infer whether 'intent_name' is a UUID, slug, or free text.
No output schemas documented. lookup_sop returns SopPolicyOut, create_case returns CreateCaseOut, but the response structure is not visible in tool_specs.py. LLMs cannot plan downstream calls without knowing what fields are returned.
No error handling or recovery guidance. Tools do not document failure modes, retryability, or what the LLM should do if lookup_sop returns no policy or create_case fails due to database constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Tool descriptions are too brief (30-50 chars). 'Fetch SOP policy by intent_name' lacks context on when to call this vs other tools, what SOP means, or what happens if no policy exists. Baseline for A+ tools is 50-200 chars.
No idempotency or confirmation pattern for destructive operations. create_case and append_audit modify state but lack dry-run, confirmation, or idempotency hints. Agents may accidentally create duplicate cases or audit events on retry.