Loading ToolBench
MCP server for gathering SIGMET data from AviationWeather.gov API
The server defines two tools with reasonable parameter schemas and reasonable descriptions. Both tools follow the verb_noun naming convention (get_*) and include JSON Schema with type constraints and enums. However, there are notable gaps: parameter descriptions are minimal (1-2 sentences), output schemas are not explicitly documented, and error handling guidance is absent. The implementation uses Zod for validation but does not propagate validation errors back to the LLM in an actionable format. Parameter descriptions lack context about when to use each filter and what the human-readable output contains. The server lacks any toolAnnotations (destructiveHint, idempotentHint, readOnlyHint) despite both tools being read-only operations.
Retrieve domestic SIGMETs (Significant Meteorological Information) for the United States. These are weather advisories for potentially hazardous conditions affecting aircraft operations.
Retrieve international SIGMETs (Significant Meteorological Information). These are weather advisories for potentially hazardous conditions affecting aircraft operations outside the United States.
Parameter descriptions are minimal and lack context. 'Filter by hazard type: conv (convective), turb (turbulence), ice (icing), ifr (instrument flight rules)' is good, but 'Flight level ±3000 feet to search' does not explain what happens if the flight level does not match any data, or whether the ±3000 feet is a server-side constraint or a guideline. Missing guidance on when humanReadable=true vs false is useful.
Output schema is not documented. The tools return arrays of SIGMET objects, but the caller (and LLM) cannot see what fields are in each object, their types, or their meanings. This forces the LLM to reason about the structure blindly and risks extraction errors. The test file shows JSON.stringify() of results but does not define the schema formally.
Tool annotations are missing. Both tools are read-only (risk: READ_ONLY), but they do not include the readOnlyHint annotation in the tool definition. This makes it difficult for the MCP client to surface confidence about which tools are safe to call without user confirmation.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 61 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 43 | 0.4.0+ | v1 |
Error handling does not guide recovery. The tests show Zod validation errors are caught, but the tool call handler (src/index.ts snippet is truncated) does not show how validation errors are returned to the caller. If an LLM passes level=1000 (outside the 0 - 600 range), the error message must be 'Invalid level: got 1000, must be between 0 and 600', not a stack trace.
All parameters are optional, but the schema does not explain the default behavior when a parameter is omitted. If no hazard is specified, does the tool return all hazard types? If no date is specified, does it default to the current UTC time? This ambiguity forces the LLM to guess.