Loading ToolBench
Weather MCP Server providing access to US National Weather Service data
Two tools with basic schemas and descriptions, but significant gaps in error handling, output documentation, and parameter constraints. Tool names follow verb_noun pattern (get_alerts, get_forecast) which is good. Descriptions are present but minimal (18-27 chars, well below the 50-200 char LLM-optimized baseline). Input schemas use Zod with type constraints (latitude/longitude bounds, state length), but output schemas are not documented. Error responses are generic text without recovery guidance. No tool annotations (readOnlyHint, idempotentHint) despite both being read-only operations.
Get weather alerts for a state
Get weather forecase for a location
Output schemas not documented. LLMs cannot infer what fields to expect from responses (e.g., does get-alerts return event, severity, headline as separate fields or nested?). Agents cannot plan downstream tool calls or extract structured data reliably.
Tool descriptions are too short (18-27 chars vs. 50-200 char baseline). 'Get weather alerts for a state' and 'Get weather forecase for a location' lack context on WHEN to use each tool, what data is returned, or prerequisites (e.g., 'get-forecast only supports US coordinates via NWS API').
Error responses are generic text without recovery guidance. When NWS API fails, responses say 'Failed to retrieve alerts data' or 'This location may not be supported by the NWS API' but do not guide the LLM on what to do next (retry, try different coordinates, ask user for clarification).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
No tool annotations despite both tools being read-only. Adding readOnlyHint=true signals to agents that these calls are safe to retry and have no side effects, improving agent planning and error recovery.
Parameter descriptions lack format/constraint details. 'Two-letter state code (e.g. CA, NY)' includes an example (anti-pattern: LLMs reuse examples literally). Should state 'Two-letter US state code (uppercase or lowercase, e.g., CA, NY, TX)' and rely on Zod length(2) constraint, not description text.