Loading ToolBench
Multi-server MCP setup combining a weather tool server with LangChain-based chatbots (Ollama and OpenAI variants) that support persistent memory and MCP tool integration
Single tool with acceptable naming and reasonable description, but significant gaps in parameter documentation and output schema specification. The tool follows a clear verb_noun pattern (get_current_weather) and has a brief description. However, input parameters lack individual descriptions (unit parameter has none), and there is no documented output schema. The tool is READ_ONLY and safe, but error handling is minimal, the code does resp.raise_for_status() which will throw on API failures without actionable recovery guidance for the LLM.
Get current weather information for a location
Output schema not documented. The tool returns a dict with location, temperature, unit, and forecast fields, but no schema is provided in the code or documentation. LLMs cannot plan downstream operations or validate response structure.
Parameter 'unit' lacks a description in the function signature or docstring. While the description states it controls temperature unit and defaults to celsius, there is no inline parameter documentation visible in the source. LLMs cannot infer that only 'celsius' or 'fahrenheit' are valid.
Missing enum constraint on 'unit' parameter. The code hardcodes a unit_map {'celsius': 'metric', 'fahrenheit': 'imperial'}, but the schema does not declare this as an enum. LLMs may pass invalid values like 'kelvin' or 'rankine', which the tool silently ignores or miscasts.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 6 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 41 | - | v1 |
Error handling exposes implementation details without recovery guidance. API errors call resp.raise_for_status(), which throws raw HTTP exceptions. LLMs receive stack traces like 'HTTPError: 401 Unauthorized' with no actionable guidance, should say 'API key invalid or expired. Check OPENWEATHER_API_KEY environment variable.'
API key exposed as environment variable without validation or safety wrapper. While not technically a parameter, the code calls ensure_env('OPENWEATHER_API_KEY'), and if missing, raises RuntimeError. A production tool should validate key format and provide clear setup instructions, not fail at runtime during an agent call.
No timeout or rate-limit handling visible. The requests.get() call has a 15-second timeout, which is good, but no retry logic or backoff is shown. If OpenWeather API is slow or throttled, the tool will hang, blocking the agent.
Response may include unexpected fields or missing data. The code safely extracts 'main', 'weather', and 'name' with defaults, but does not validate that the API response contains the expected structure. If OpenWeather changes schema, silent failures could occur (e.g., temp is None if 'main' is missing).