Loading ToolBench
A weather service MCP server that provides weather alerts and forecasts from the National Weather Service API
This weather server has moderate tool definitions but significant gaps in schema completeness, parameter descriptions, and output documentation. Both tools have verb-noun naming (get_alerts, get_forecast) which is good. Descriptions exist but are minimal (17 and 28 chars respectively). Tool implementations show async error handling and proper HTTP wrapping, but schemas lack critical details. Parameters lack validation constraints, ranges, and detailed descriptions beyond the bare type. Output is unstructured free-text strings rather than typed JSON objects, making downstream parsing error-prone for agents. No pagination support despite APIs that could return many results. No per-field output schema documentation.
Get weather alerts for a US state.
Get weather forecast for a location.
Minimal tool descriptions (17-28 chars) lack actionable context. 'Get weather alerts for a US state' does not explain when to use this vs forecasts, what the return format is, or error conditions. Baseline for A-grade tools is 50-200 chars with clear selection guidance.
Parameter 'state' lacks validation constraints. Description says 'Two-letter US state code (e.g. CA, NY)' but does not enforce it: no enum, no regex pattern, no length constraint. LLM could pass 'California', 'usa', or 'XX' and fail at runtime.
Parameters 'latitude' and 'longitude' lack bounds or validation description. No min/max, no precision guidance (degrees vs radians?), no invalid-value examples. LLM could pass latitude=200 or invalid formats.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 40 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 46 | - | v1 |
Output is unstructured plain text strings, not typed JSON. 'No active alerts for this state' or multi-line formatted strings force agents to parse natural language, error-prone and wasteful. Should return structured arrays of alert/forecast objects with typed fields (event, severity, temperature, etc.).
No output schema documentation. LLMs do not know what fields to expect from responses. Should document: for get_alerts, returns list of {event, area, severity, description, instructions}; for get_forecast, returns list of {name, temperature, windSpeed, windDirection, detailedForecast}.
get_forecast hardcodes result limit to 5 periods with no mention in description. Users cannot request more data or paginate. Should expose a limit parameter (1-100, default 5) and document it.
Error handling returns generic strings ('Unable to fetch alerts or no alerts found', 'Unable to fetch forecast data') that do not guide recovery. Missing guidance: is this a temporary network issue (retry)? Invalid coordinates (user must provide valid location)? API service down (wait)? LLM cannot distinguish.
No error categorization. Responses do not indicate whether failures are retryable (transient network timeout), user-fixable (invalid state code), or fatal (API rate-limited). Agent cannot make intelligent retry decisions.