Loading ToolBench
Weather MCP Server that provides real-time weather information by querying OpenWeatherMap API
Single tool 'get_weather' has a partial schema (city parameter present with Zod validation) and a description, but multiple critical quality issues prevent a higher score. The tool description is actually a PARAMETER description misplaced at the tool level, not a description of what the tool does. Parameter 'city' is described generically without format guidance. No output schema is documented for the LLM. The tool lacks error classification, recovery guidance, and does not follow agentic patterns for tool composition. No tool annotations (readOnlyHint, etc.) despite being READ_ONLY. The implementation internally translates city names and calls OpenWeatherMap API, but the interface does not expose this clearly to the agent.
The name of the city. This will be translated to English before querying the weather service.
Tool description is missing entirely. The string at tool level ('The name of the city...') is actually the description of the 'city' parameter, not the tool itself. LLMs cannot determine WHAT this tool does, WHEN to use it, or what it RETURNS without a proper tool description.
No documented output schema. The tool returns a JSON object with fields (city, temperature, condition, humidity, wind_speed, country) but the LLM has no advance knowledge of this structure. LLMs cannot plan downstream usage or extract fields without explicit schema documentation.
No error recovery guidance. Error responses are JSON strings with an 'error' field, but do not guide the LLM on next steps. For example, 'Could not find weather data for [city]' should suggest 'Try a different city name or spelling' or offer available cities. Pattern: recovery-guide.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 40 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 47 | - | v1 |
No tool annotations despite READ_ONLY risk profile. The tool should include toolAnnotations with readOnlyHint=true to signal to agents that this is a safe, repeatable operation with no side effects.
Parameter description lacks format/constraint guidance. 'The name of the city. This will be translated to English before querying the weather service.' does not explain: What happens if the city name is ambiguous (e.g., 'Springfield' in 50 US states)? What language input is accepted? What format (e.g., 'City, Country' or just 'City')? Specification should be actionable: '2-50 character city name. Optionally include country code (e.g., "London, UK") to disambiguate.'
Tool composition issue: translation logic is internal and hidden. The tool promises to translate city names to English, but this is an implementation detail not exposed in the interface. If an agent wants to see what 'Tokyo' translates to, or handle translation failures explicitly, it cannot. Consider exposing translation as a separate tool or making the translation step visible in error messages.
Error handling does not differentiate error types. A 404 (city not found), 401 (invalid API key), timeout, and network error all return unstructured JSON strings. Errors should be categorized: 'user_fixable' (try a different city), 'retryable' (temporary API outage), or 'fatal' (invalid API key). This enables smarter agent behavior.