Loading ToolBench
Single tool with adequate schema but significant gaps in description quality and parameter documentation. Tool name 'get_weather' follows verb_noun convention (+). Input schema is present with enum constraint on city parameter (+). However, tool description is only 24 characters in English (translated from Japanese), below the 34-char p10 baseline and insufficient for LLM selection reasoning. No output schema documented. Parameter 'city' has a description but lacks guidance on what happens when an unsupported city is passed. Error handling exists but is generic. No tool annotations (readOnlyHint, etc.) despite being a read-only operation.
指定された都市の天気情報を取得します
Tool description is too short (24 chars). Baseline p10 is 34 chars; A+ tools average 194 chars. Current description 'Get weather information for a specified city' (translated) does not explain WHEN to use this tool, what it returns, or how it differs from similar tools.
No output schema documented. LLMs cannot plan downstream operations or extract fields without knowing the response structure. Current code returns JSON with city, weather, temperature fields, but this is not declared in tool metadata.
Missing tool annotations. Tool is read-only but does not declare readOnlyHint. This forces LLMs to infer safety from the name alone, increasing hallucination risk for destructive operations in multi-tool scenarios.
Error message for unsupported city is generic ('City not found: {city}'). Should include available alternatives: 'City not found: {city}. Available cities: 東京, 大阪, 福岡, ロンドン, シアトル, パリ, ニューヨーク' to enable LLM self-correction.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | 2026-07-28+ | v2 |
Parameter 'city' description lacks format guidance. Should state: 'City name (must be one of the enum values; case-sensitive Japanese or English names)' to prevent LLM from attempting variations like 'Tokyo' instead of '東京'.