Loading ToolBench
AI agent that researches 100 software applications to determine their buildability as AI-agent-callable toolkits. Uses a two-pass verification loop: Pass 1 extracts structured records from web search snippets; Pass 2 fetches actual documentation and verifies/corrects findings.
Three tools with mixed quality. search and fetch_page_text have reasonable descriptions (100+ chars) and basic input schemas with types and descriptions. record_app has a comprehensive schema with 15 fields and enums, but descriptions are minimal or missing for most parameters. No output schemas documented. Error handling is absent, tools return raw strings like '[FETCH_FAILED: ...]' without recovery guidance. No tool annotations (readOnlyHint/destructiveHint). Naming is verb-forward (search, fetch, record) but descriptions lack WHEN/WHY context. Parameter descriptions exist but are sparse (e.g., 'Application name' for 'app' param). No pagination, no batching, no idempotent guarantees stated.
Best-effort raw fetch + text extraction for a docs page. Strategy: 1. First try Tavily extract (renders JS, better for SPAs) 2. Fall back to direct HTTP fetch with HTML stripping 3. Falls back gracefully — some docs sites block bots, agent should treat empty result as low confidence, not crash.
Record structured research findings for one app.
Returns list of {title, url, content} snippets.
record_app parameter descriptions are minimal or absent. 'app', 'category', 'one_liner', 'blocker', 'notes', 'evidence_url', 'mcp_evidence_url' lack descriptions explaining what values are expected, constraints, or examples. LLMs cannot infer meaning from names alone.
No output schemas documented for any tool. search and fetch_page_text return unstructured strings or lists; record_app returns nothing visible. LLMs cannot plan downstream calls or extract structured data without knowing response shape.
Error handling is non-actionable. fetch_page_text returns '[FETCH_FAILED: ...]' strings that do not guide the LLM on recovery. No error classification (retryable vs fatal), no suggestions for next steps, no invalid-value feedback.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
No tool annotations. search and fetch_page_text are READ_ONLY; record_app is WRITE. These should be declared via readOnlyHint/destructiveHint in the tool definition so agents know which calls are safe to retry and which have side effects.
Descriptions lack WHEN/WHY context. 'Returns list of {title, url, content} snippets' does not explain when to call search vs fetch_page_text, or what the agent should do with results. Descriptions should guide tool selection.