Loading ToolBench
Multi-agent system for research and report generation with MCP server integration for documentation lookup
Five tools with mixed quality. web_search, compute_metric, and docs_lookup have reasonable descriptions (50-100 chars) and basic schemas, but lack parameter descriptions and output schema documentation. save_report has a good description but no input schema visible in the code. list_docs has minimal description (34 chars, below baseline 194). No tool includes error recovery guidance, output field documentation, or parameter constraints (enums, ranges). Naming is verb-forward (good), but schemas are incomplete and descriptions lack LLM-optimization details like when to use each tool or what to expect in responses.
Compute a simple aggregate metric over a list of numbers.
Return the contents of the doc most relevant to `topic`. Tries exact stem, then stem substring, then a content search that ranks docs by topic-word overlap so callers don't need exact filenames (e.g. "Runner" -> agents).
List the available documentation topics (file stems).
Persist the final report to disk. Pauses for human approval before writing.
Search the knowledge base, ranking entries by query-word overlap.
Parameter descriptions missing. 'query' in web_search, 'operation' and 'values' in compute_metric, 'topic' in docs_lookup lack descriptions explaining constraints, formats, or valid values. LLMs cannot infer parameter meaning from names alone.
Output schemas not documented. No tool specifies what fields are returned or their types. web_search returns a string; compute_metric returns a formatted string; docs_lookup returns markdown text. LLMs need structured output documentation to plan downstream calls.
No error recovery guidance. compute_metric returns 'ERROR: percent_change undefined when the first value is 0' and docs_lookup returns 'NO_DOC: no doc matched...' but neither tells the LLM what to do next. Should suggest alternative tools or corrective actions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
list_docs description is 34 characters ('List the available documentation topics (file stems).'), below the 194-char baseline. Lacks context on when to call it or what structure it reveals. Should explain: 'Call this first to discover available documentation topics before using docs_lookup.'
No input constraints or enums. compute_metric accepts 'operation' as a string but should declare it as an enum (sum|mean|percent_change). web_search 'max_results' has no min/max bounds. Unconstrained inputs invite hallucinated values.