Loading ToolBench
Four tools with complete input schemas and basic descriptions, but significant gaps in parameter documentation and output schema clarity. Tool names follow verb_noun convention (echo, add_numbers, list_items, deploy), which is good. However, descriptions are minimal (10-60 chars) and lack context about when to use each tool or what they return. Parameters have type definitions but most lack descriptions beyond the name. No output schemas documented. Error handling is absent, tools return generic TextContent with no guidance for LLM recovery. The 'deploy' tool is marked WRITE but has no confirmation or dry-run pattern. Overall, definitions are skeletal, sufficient for basic invocation but insufficient for reliable agent planning.
Add two numbers
Deploy to an environment (shadows global --env and --refresh)
Echo back the input
List items in a directory (test tool)
Tool descriptions are too brief (10-60 chars) and lack context. 'Echo back the input' does not explain when to use echo vs other tools, what it returns, or whether it has side effects. LLMs cannot reliably select tools without richer descriptions (target 50-200 chars).
Most parameters lack descriptions. 'message', 'a', 'b', 'path', 'recursive', 'env', 'refresh' have no parameter-level documentation. LLMs cannot infer whether 'path' is absolute or relative, whether 'recursive' defaults to false, or what 'env' values are valid.
No output schemas documented. Tools return TextContent with unstructured text. LLMs cannot plan downstream calls or extract structured data. E.g., list_items should return an array of file objects with name, type, size; add_numbers should return a numeric result.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
No error handling or recovery guidance. Tools return generic 'Unknown tool' or 'Resource not found' messages with no actionable next steps. LLMs cannot self-correct or retry intelligently.
The 'deploy' tool is marked WRITE (destructive) but has no confirmation, dry-run, or idempotent hint. Agents may invoke it without understanding consequences. Missing tool annotations (destructiveHint, idempotentHint).