Loading ToolBench
Two tools with clear verb-noun naming (get-alerts, get-forecast) and basic descriptions. Both have input schemas with type constraints (string length, numeric bounds). However, descriptions are minimal (under 50 chars), lack context on when to use each tool, and do not explain prerequisites or return structure. No output schema documentation. Error handling returns plain text without recovery guidance. Parameters have descriptions but lack detail on format expectations. No tool annotations (readOnlyHint, etc.). Schemas are present but sparse, missing structured output definitions.
Get weather alerts for a state
Get weather forecast for a location
Descriptions are too brief (under 50 chars each). 'Get weather alerts for a state' lacks context on when to use this vs get-forecast, what data is returned, or prerequisites (e.g., state code format). Baseline for A+ tools is 50-200 chars with WHAT/WHEN/WHY.
No output schema documented. LLMs cannot plan downstream calls or extract fields without knowing the response structure. Both tools return text content but do not declare field names, types, or structure.
Error responses are plain text without recovery guidance. 'Failed to retrieve alerts data' tells the LLM nothing, should be 'Failed to retrieve alerts for state {state}. Verify the state code is a valid 2-letter US state abbreviation (e.g., CA, NY). If the issue persists, the NWS API may be temporarily unavailable.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
No tool annotations. Both tools are read-only (risk: READ_ONLY declared in metadata) but lack readOnlyHint in schema. This prevents clients from optimizing caching or parallelization.
Parameter descriptions lack format details. 'Two-letter state code (e.g. CA, NY)' includes example values (anti-pattern: LLMs reuse examples literally). Should use enum constraint or pattern. Latitude/longitude descriptions omit coordinate system (WGS84 assumed but not stated).