Loading ToolBench
A demonstration MCP server with weather alerts and forecast tools, plus basic arithmetic and greeting resources
Three tools with basic schemas and descriptions, but significant gaps in parameter documentation and output structure. The 'add' tool is trivial (integer arithmetic). Weather tools have reasonable descriptions but lack output schema documentation, parameter constraints (e.g., latitude/longitude bounds), and error recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Parameter descriptions exist but are minimal (10-40 chars vs. baseline 72 chars). No pagination, no structured output schemas, no error classification.
Add two numbers
Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY)
Get weather forecast for a location. Args: latitude: Latitude of the location longitude: Longitude of the location
Output schemas not documented. Tools return free-text strings (e.g., 'Unable to fetch alerts or no alerts found') instead of structured objects. LLMs cannot parse results reliably or chain to downstream tools.
Parameter constraints missing. Latitude/longitude have no bounds (±90, ±180), state code has no enum or pattern validation. LLMs can pass invalid values (latitude=999) without guidance.
Error handling provides no recovery guidance. 'Unable to fetch alerts' and 'Unable to fetch forecast data' are bare failures with no actionable next steps for the LLM (retry? call different tool? ask user?).
No tool annotations. Tools lack readOnlyHint, destructiveHint, or idempotentHint metadata. Agents cannot determine retry safety or side-effect scope.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
Parameter descriptions are minimal (10-40 chars vs. baseline 72 chars). 'Two-letter US state code (e.g. CA, NY)' includes example values that LLMs may reuse literally; should use enum constraint instead.