Loading ToolBench
The server defines 3 tools with basic schemas and descriptions, but exhibits significant quality gaps. All tools have names starting with action verbs (get_, summarize_, query_) and non-empty descriptions, meeting minimal requirements. However, descriptions are brief (27-77 characters), parameter descriptions exist but are minimal, and there is no documented output schema for any tool. Tools return plain strings rather than structured objects. Error handling is present in helpers but not propagated as actionable LLM guidance. The tool definitions are directly visible in server.py via mcp.tool() decorators, so they are not inferred. This represents a typical community server: functional but not production-ready for agent use.
Fetches real-time weather for a city using an external API.
Executes an SQL query on the tasks database and returns a summary.
Summarizes a CSV file by returning its row count, columns, and first row.
No documented output schemas for any tool. All tools return plain strings (str) rather than structured objects. LLMs cannot plan downstream calls or extract specific fields from responses. Violates pattern:response-shaper and pattern:tool requirements.
Tool descriptions are under 100 characters and lack context about when to use each tool vs alternatives. 'Summarizes a CSV file...' (52 chars) does not explain prerequisites (file must exist in data/), expected use cases, or relation to query_tasks.
Parameter descriptions are generic. 'The name of the CSV file to summarize' does not specify format, path prefix (data/), allowed characters, or max length. 'The SQL query to execute' does not document allowed tables, query complexity limits, or injection risks.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | 2026-07-28+ | v2 |
query_tasks accepts arbitrary SQL without validation visible in helpers.py. No parameter enum, no mention of allowed tables, no injection safeguards visible. Agents could attempt DROP TABLE or SELECT from forbidden tables. Violates pattern:tool-gateway and pattern:secret-injection (database connection string should not be exposed).
Error responses in helpers.py are plain strings ('Error summarizing file: {str(e)}') with no actionable recovery guidance. Per pattern:recovery-guide, errors must tell the LLM what to do next, categorize as retryable/user-fixable/fatal, and include the invalid value and constraint violated.
get_weather calls OpenWeather API and exposes OPENWEATHER_API_KEY via os.getenv(). While the key is not a tool parameter (correct), there is no rate limiting, timeout explicit in the code, or per-user quota tracking. An agent in a retry loop could exhaust daily API limits. Per pattern:scope-declaration and pattern:secret-injection, rate limiting and clear quota communication are required.
summarize_csv_file parameter 'filename' is a bare string with no validation. Agents could pass paths like '../../../etc/passwd' (path traversal). Code prefixes with 'data/', but no validation that filename contains only safe characters. Per pattern:tool-gateway, all agent input must be sanitized.
No pagination or result limits on query_tasks. An agent querying a large table with SELECT * could fetch millions of rows, exhausting context window and API tokens. Per pattern:paginated-result and mxe:enforce-result-limits, tools returning lists must accept limit/offset and document max result size.
summarize_csv_file and query_tasks both operate on data but serve different purposes (file vs database). Tool names are clear, but descriptions do not explain when to use one vs the other. Per pattern:tool-chain composition guidance, related tools should guide LLM selection via descriptions.