Loading ToolBench
A Pydantic AI agent-based application for customer support with banking domain logic
Single tool with basic schema and description. Tool name 'customer_balance' lacks action verb prefix (should be 'get_customer_balance'). Description is adequate (89 chars, within 10-1024 range) but lacks context on when to use it vs alternatives. Input schema present with type and description for 'include_pending' parameter. No output schema documented. No error handling guidance, ValueError('Customer not found') is raised but not described in tool docs. Tool is read-only (good), but no explicit idempotent hint. Overall: functional but below production baseline (194 char avg description, 100% of A+ tools document output).
Retrieves the customer's account balance, optionally including pending transactions
Tool name lacks action verb prefix. 'customer_balance' should be 'get_customer_balance' to signal retrieval intent to LLM.
No output schema documented. LLM cannot infer return type (float | None) or plan downstream operations. Baseline: 100% of A+ tools document return types.
Error handling not documented. ValueError('Customer not found') is raised but tool description does not explain error conditions or recovery steps (e.g., 'If customer not found, try search_customers first').
Tool description lacks context on when to use it. Should explain: What does it do? When should the LLM call it? What does it return? Current description is minimal.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
No tool annotations present. Tool is read-only but lacks readOnlyHint annotation to signal safety to LLM. Idempotent operations should declare idempotentHint.