Loading ToolBench
Single tool (calculator_tool) with complete input schema and clear description. Schema is well-formed with typed parameters (operation enum, numeric a/b). Description is concise (79 chars, within 10-1024 baseline). However, output schema is not documented, LLMs cannot see what fields to expect from the response. No error handling guidance (e.g., division by zero is caught but error message is generic). Tool name lacks action verb prefix (should be 'calculate' or 'perform_calculation'). No parameter descriptions visible in schema, only type constraints. Missing tool annotations (readOnlyHint, idempotentHint) despite being READ_ONLY and idempotent.
Performs basic arithmetic operations: add, subtract, multiply, divide.
Tool name lacks action verb. 'calculator_tool' does not start with a verb (get_, create_, perform_, calculate_). LLMs infer intent from the name before reading description. Rename to 'calculate' or 'perform_calculation'.
Output schema not documented. Tool returns {result: number} but this is not visible in the tool definition or description. LLMs cannot plan downstream calls or extract the result field without guessing.
Parameter descriptions missing from schema. Input parameters (operation, a, b) have no descriptions in the JSON Schema, only type constraints. LLMs rely on descriptions to understand what each parameter controls.
No tool annotations. Tool is READ_ONLY and idempotent but lacks readOnlyHint and idempotentHint annotations. Agents cannot determine safety/retry semantics without explicit hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
Error handling lacks recovery guidance. Division by zero throws HttpError(400, 'Division by zero is not allowed.') but does not suggest what the LLM should do next (e.g., 'Try with b != 0').