Loading ToolBench
Weather MCP server providing access to US National Weather Service alerts and forecasts
This server has functional tool definitions with basic descriptions and schemas, but falls short of production quality. Both tools (get_alerts, get_forecast) use action verbs correctly and have input schemas with types. However, descriptions are minimal (17-26 characters), parameter descriptions are vague or missing, output schemas are undocumented, and error handling is weak. The rubric baseline for tool descriptions is 194 chars (p10=34, p90=392); these descriptions at 17-26 chars fall below the p10 threshold. Parameter descriptions exist in the code docstrings but are not integrated into the schema or explicitly visible to the LLM. No output documentation tells the agent what structure to expect. Error responses are plain strings without recovery guidance. The server follows basic naming conventions and has inputs, but lacks the LLM-optimized descriptions and comprehensive schema detail required for A/B tier grades.
Get weather alerts for a US state.
Get weather forecast for a location.
Tool descriptions are too short (17-26 chars) to guide LLM selection. Rubric baseline is 194 chars; these descriptions fall below p10=34. 'Get weather alerts for a US state' and 'Get weather forecast for a location' provide no context for WHEN to use the tool, what data structure is returned, or how it differs from related tools.
Output schemas are completely undocumented. Both tools return plain strings, but the agent does not know: (1) what fields the string contains, (2) whether it is structured or freeform, (3) how to parse it for downstream operations. LLMs need to know what fields to expect so they can plan downstream tool calls and extract the right data.'
Parameter descriptions in code docstrings are not visible in tool schemas or FastMCP registration. FastMCP tool decorator does not expose parameter-level descriptions to the protocol. 'latitude' and 'longitude' parameters have no description in the MCP schema, forcing the LLM to infer meaning from the name alone. This violates the rule: 'Every parameter needs a description explaining what it controls.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 42 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 44 | - | v1 |
Error handling returns plain strings ('Unable to fetch alerts...', 'No active alerts...') with no recovery guidance. A raw error code or stack trace gives the agent nothing to act on.' When alerts fail, should the agent retry? Check the state code? Call a different tool? No guidance provided.
No input validation or constraint documentation. 'state' parameter accepts any string, but only two-letter US state codes are valid. No enum constraint, no format validation, no error message if an LLM passes 'California' or 'invalid'. Rubric: 'When a parameter accepts one of a known set of values, declare it as an enum.'
Latitude/longitude parameters lack range constraints. What happens if an LLM passes latitude=999 or longitude=null? No validation, no error bounds. Rubric: 'Specify minimum and maximum for numeric parameters (e.g. page_size 1 - 100, days 1 - 365).'
get_forecast returns up to 5 forecast periods hardcoded ([:5]), but no pagination or limit parameter. If the user wants all periods, or just the next 3 hours, there's no way to control this. Rubric: 'Tools returning lists should accept page/offset and limit parameters.'
Responses return verbose NWS API fields (temperature, temperatureUnit, windDirection, detailedForecast) as freeform strings. Rubric: 'Return structured objects with typed fields. Free-text responses require the LLM to parse unstructured output, which is error-prone and wastes tokens.' A structured JSON response with forecast array of {period, temperature, wind, forecast} objects would be far more composable.