Loading ToolBench
Single tool 'define_behavior' has a well-structured schema with proper types and descriptions. The tool description is comprehensive (280+ chars) and explains WHAT it does, WHEN to use it, and what it returns. All four parameters have descriptions and appropriate constraints (enums for 'side' and 'depth', array for 'domains'). However, the tool lacks output schema documentation, the description says it 'Returns curated cases...as a sectioned spec' but the actual response structure is not formally documented. This forces LLMs to infer the output shape from the JSON response alone. The tool name 'define_behavior' is clear and verb-forward, though 'generate_behavior_spec' or 'probe_behavior_spec' might be more action-oriented. No error handling guidance is visible in the code, invalid feature_description triggers a generic 'feature_description is required' error with no recovery hints.
Produce the behaviour specification for a feature BEFORE writing code: what it must do, the contract it must honour, the boundaries it must hold at, the conditions it must survive, and the guarantees it must not break. Returns curated cases from the prespec bank as a sectioned spec, the open questions to put to the user, and the defaults assumed on their behalf. Call this at the start of any feature touching a covered domain — the spec becomes the acceptance criteria the tests and the code are written against.
Output schema not documented. Tool description says it returns 'curated cases...as a sectioned spec' but the actual response structure (fields, types, nesting) is not formally specified. LLMs must infer structure from runtime responses.
Error handling lacks recovery guidance. Invalid 'feature_description' returns 'feature_description is required' with no hint on what constitutes valid input or what to do next. Per pattern:recovery-guide, errors should guide the LLM toward resolution.
Tool name 'define_behavior' is clear but not action-verb-forward. Alternatives like 'generate_behavior_spec' or 'probe_behavior_spec' would better signal the action taken. Current name reads more like a noun phrase.
Parameter 'domains' lacks guidance on valid domain paths. Description says 'e.g. backend/rest-api/pagination' but does not enumerate available domains or explain discovery. LLMs may hallucinate invalid domain paths.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
No pagination or result-limiting guidance. Tool description does not specify max cases returned or whether results are paginated. For 'deep' depth (25 cases), response could be large; no guidance on handling or truncation.