Loading ToolBench
An MCP server that provides weather alerts and forecast data using the National Weather Service (NWS) API
This server exhibits significant quality gaps. While tool names follow verb-noun conventions (get_alerts, get_forecast, get_weather_alerts, get_app_config, get_greeting), descriptions are present but minimal. The critical issue is that input schemas are either absent or incomplete in the visible source code. The server defines 5 tools, but the source is fragmented across multiple files and the schema definitions for several tools are not fully visible (particularly get_app_config and get_greeting). The get_app_config tool has an empty input schema '{}' which is valid but suggests minimal parameterization. Parameters that are present (state: string, latitude/longitude: float, name: string) have descriptions, but lack comprehensive constraint documentation (e.g., no validation that state must be a 2-letter code, no bounds on latitude/longitude). Tool descriptions are adequate but generic, they describe what the tool does but lack context on when to use it relative to similar tools (get_alerts vs get_weather_alerts are nearly identical). No output schemas are documented, no pagination guidance for potentially large alert lists, no error recovery instructions, and no tool annotations (readOnlyHint, destructiveHint, idempotentHint). The server is STDIO+SSE capable but source shows SSE transport.
Get weather alerts for a US state.
Static configuration data.
Get weather forecast for a location.
Get a personalized greeting
Get active weather alerts for a given state.
Duplicate/overlapping tool functionality: get_alerts and get_weather_alerts both fetch weather alerts for a state with identical input/output signatures. LLMs cannot disambiguate which to call, wasting reasoning cycles and risking wrong tool selection.
Missing output schema documentation. Tools return formatted strings (alerts, forecasts, greetings) but the response structure is not formally documented. LLMs cannot plan downstream operations or validate response data without knowing the output shape.
No input validation constraints documented. The 'state' parameter must be a 2-letter US state code but this is mentioned only in the description text, not in JSON Schema constraints (no enum, no pattern, no minLength/maxLength). Similarly, latitude/longitude lack bounds (must be -90 to 90, -180 to 180). LLMs cannot validate inputs before calling.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 51 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 37 | - | v1 |
No error recovery guidance. Error handling exists (make_nws_request catches exceptions and returns None), but tools return generic strings like 'Unable to fetch alerts' or 'No alerts found' without actionable next steps. No indication of retryability, no suggestions for alternative tools or manual intervention.
Incomplete source visibility. The get_app_config and get_greeting tool definitions are cut off in the provided source (weather.py ends mid-function). This prevents full assessment of their schemas and implementations.
No pagination or result limits documented. get_forecast hardcodes limit to next 5 periods (good), but get_alerts could return many items with no documented limit, pagination support, or guidance on token cost. Large alert lists risk context window exhaustion.
Missing tool annotations (readOnlyHint, destructiveHint, idempotentHint). All tools are read-only and idempotent, but these semantic hints are not declared in the tool definitions. Agents cannot infer safety properties without explicit annotation.
Descriptions lack LLM-optimization context. Descriptions state what the tool does but not WHEN to use it vs similar tools. For example, 'Get weather alerts for a US state' doesn't explain that this is equivalent to get_weather_alerts, or what domain-specific use case each serves. Baseline from rubric: optimal descriptions are 50-200 chars with clear intent and selection criteria.