Loading ToolBench
Three tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. All tools have descriptions (10-60 chars, below the 50-200 char baseline for LLM optimization). Input schemas present but incomplete: 'method' in scan_api lacks enum constraint (GET|POST|PUT|DELETE); 'status' in list_scans lacks enum despite mentioning valid values in description. No output schemas documented. Error handling exists but lacks recovery guidance. Tool names follow verb_noun pattern (scan_api, get_scan, list_scans) which is good, but descriptions are too terse to guide LLM selection effectively.
Get results of a previous middleBrick scan by its ID
List previous middleBrick API security scans
Scan an API endpoint for security vulnerabilities and get a risk score
Output schemas not documented. LLMs cannot plan downstream calls or extract required fields (e.g., scanId from list_scans to pass to get_scan). Formatters exist but their output structure is not declared in tool definitions.
Parameter 'method' in scan_api accepts free-form strings with no enum constraint. Description says '(default: GET)' but does not list valid HTTP methods. LLMs may hallucinate invalid methods like 'FETCH' or 'RETRIEVE'.
Parameter 'status' in list_scans mentions valid values in description ('queued, processing, completed, failed') but lacks enum schema. Free-form string invites invalid values; enum constraint is self-documenting.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
Descriptions are 35-40 characters, well below the 50-200 char baseline for LLM optimization. 'Scan an API endpoint for security vulnerabilities and get a risk score' lacks WHEN to use it, prerequisites, or what happens on retry. Descriptions do not answer: what does it return? When should the LLM call it instead of a similar tool?
Error handling in handleToolCall returns generic MiddleBrickError messages without recovery guidance. 'Invalid or expired API key' is actionable, but other errors like 'Unknown tool' or raw error.message do not tell the LLM what to do next (retry? ask user? call a different tool?).