Loading ToolBench
A weather information server that provides weather alerts and forecasts for US locations using the National Weather Service API
Two tools with minimal descriptions, no documented output schemas, no input validation guidance, and missing parameter constraints. Both tools are READ_ONLY (good) but definitions lack production-grade rigor. Schema definitions are present via attributes but output formats are entirely undocumented. Tool names are action-verb compliant ('Get*'), but descriptions are generic (8-40 words). No error handling guidance, no pagination, no field documentation for composite responses.
Get weather alerts for a US state.
Get weather forecast for a location.
No output schema documented for either tool. GetAlerts and GetForecast return formatted strings; LLMs cannot parse the response structure or extract specific fields for downstream tool calls. Baseline: 100% of A+ tools document return types.
Descriptions are generic and lack action context. 'Get weather alerts for a US state' (42 chars) and 'Get weather forecast for a location' (37 chars) do not explain WHEN to call them, what makes them distinct, or what the response contains. Baseline: avg 194 chars, p10=34, p90=392. These are at p10 and provide minimal guidance.
No input validation or constraint documentation. 'state' parameter in GetAlerts has no enum of valid US states, no format guidance, and no error recovery hint. 'latitude' and 'longitude' in GetForecast have no range constraints (e.g., -90..90, -180..180) or format examples. Baseline: constrained-input pattern requires enums for bounded sets.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 36 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 35 | - | v1 |
No error handling guidance. Code calls response.EnsureSuccessStatusCode() and throws generic 'Exception' with minimal context. If the API returns 404 (invalid state) or invalid coordinates, the LLM receives a raw exception stack trace with no recovery hints. Pattern: recovery-guide requires telling the agent what to do next.
GetForecast returns detailed forecast periods as unstructured multi-line strings. No field-level parsing available to LLM. Response includes redundant wind direction, temperature units hard-coded as °F with no option for user preference. Baseline: mxe:strip-api-responses, remove noise, return structured fields only.
GetAlerts returns all active alerts for a state joined with '--' delimiter. No pagination, no limit, no count. If a state has 100+ active alerts (during severe weather), response could be enormous and blow context windows. Baseline: paginated-result pattern requires limit + offset + total count.