Loading ToolBench
Deterministic orchestration for LLM pipelines: authority in code, phrasing in the LLM. Company research is the reference example.
Four tools with complete Pydantic schemas and descriptions. Naming follows verb_noun convention (add_task, list_tasks, complete_task, remember_preference). Descriptions are adequate (40-80 chars) but lack LLM-optimization guidance on WHEN to use each tool or dependencies between them. All parameters have types and descriptions. Output schemas are documented via Pydantic models. However, descriptions are generic and do not explain prerequisites, error recovery, or multi-step composition. No error handling guidance visible in tool definitions. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear WRITE vs READ_ONLY semantics.
Add a new task, optionally with a due date.
Mark a task as done by its id.
List tasks, optionally including completed ones.
Remember a user preference or fact for future conversations.
Descriptions lack LLM-optimization guidance. No explanation of WHEN to call each tool, dependencies (e.g., 'list_tasks before complete_task'), or error recovery paths.
No tool annotations despite clear semantics: list_tasks and complete_task should have readOnlyHint=true/false; add_task and remember_preference should have destructiveHint=true.
No error handling guidance in tool definitions. What happens if task_id is invalid? What should the LLM do next?
Parameter 'due' in add_task accepts free-form string with example format. Should validate ISO 8601 or provide enum of relative dates (today, tomorrow, next_week).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | <=2025-11-25 | v2 |
Tool composition not documented. No guidance on typical workflows (e.g., 'Call list_tasks first to see available tasks, then complete_task').