Loading ToolBench
Four tools with complete JSON schemas and descriptions, but significant quality gaps. Tool names lack clear action verbs (declare_context_transition, suggest_guidance are vague). Descriptions are present but generic (10-50 chars for some params). Parameter descriptions exist but lack actionable constraints (no enums for suggestion_type values, no format specs). Output schemas are not documented, LLMs cannot predict response structure. Error handling is absent, no recovery guidance. Tools appear to conflate multiple concerns (declare_context_transition both logs and returns guidance). No tool annotations (readOnlyHint, destructiveHint). Baseline: avg tool description ~194 chars; these average ~80 chars. Param descriptions average ~40 chars vs baseline 72 chars.
Declare a context transition and receive guidance for the new context
Fetch guidance for a context (mid-context refresh or preview)
Suggest improvements to guidance based on discoveries
Vote on guidance usefulness with fuzzy matching
Tool names lack clear action verbs. 'declare_context_transition' is vague (declare vs enter vs switch?). 'suggest_guidance' conflates suggestion creation with guidance improvement. LLMs cannot infer intent from names alone.
No output schemas documented. LLMs cannot predict response structure (e.g., does declare_context_transition return guidance object, string, or array?). Breaks tool chaining and forces agents to guess.
Parameter descriptions lack actionable constraints. 'suggestion_type' has enum values but description does not explain when to use each (tool vs workflow vs gotcha). 'context' param has no format spec or example.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | <=2025-11-25 | v2 |
No error handling or recovery guidance. Tools do not document failure modes (e.g., what if context_id does not exist? What if suggestion already exists?). LLMs receive no actionable error messages.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Agents cannot determine which tools are safe to retry or have side effects. declare_context_transition and suggest_guidance are clearly destructive but unmarked.