Loading ToolBench
A collection of MCP server experiments including a project tracker tool and a PostgreSQL query assistant
This MCP server exhibits significant quality gaps across naming, descriptions, schemas, and error handling. While 6 of 7 tools have basic descriptions and visible input schemas with type definitions, the descriptions are frequently generic and lack actionable guidance for LLM tool selection. Parameter descriptions are sparse or missing entirely. Error handling is minimal, most tools lack recovery guidance or categorization. The run_query tool is a SQL injection risk. Output schemas are not documented. Tool compositions rely on numeric IDs rather than natural identifiers (e.g., project_id instead of project_name). This server is typical of early-stage MCP implementations and would not pass code review by a production tool engineer.
Add a new project with name and optional description.
Add a task to a project. category: 'daily' or 'monthly' due_date: YYYY-MM-DD
List all projects
Run a custom SELECT query on the employees table. Only SELECT statements are allowed for safety.
Attach GitHub URL to a project
Update a task's status and/or add a progress note. Status must be one of: pending, in-progress, done.
View project tasks
SQL Injection vulnerability in run_query tool. The tool accepts arbitrary SQL strings and validates only by checking if the query starts with 'select' (after lowercasing). An attacker can bypass this with comments ('SELECT /**/; DROP TABLE...') or case variations. Additionally, the tool directly executes user input via sqlalchemy.text() without parameterized queries, exposing the database to injection attacks.
Missing output schema documentation for all tools. The rubric requires documented return types for 100% of production tools. Tools like view_tasks, list_projects, and run_query return List[Dict] but the structure of those dicts (field names, types, presence) is not specified anywhere. LLMs cannot infer what fields are available or plan downstream calls reliably.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 43 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
Sparse parameter descriptions. Parameters like 'category' in add_task, 'status' in update_task, 'sql' in run_query, and 'progress_note' in update_task lack detailed descriptions. Parameter descriptions should explain format, constraints, and LLM guidance. For example, 'category' should read: 'Task category; must be either "daily" (repeating daily) or "monthly" (repeating monthly).' Currently reads 'Task category: \'daily\' or \'monthly\'', insufficient for LLM disambiguation.
Minimal error handling and no recovery guidance. Tools return plain success strings ('✅ Project \'name\' added.') or generic error messages ('❌ Category must be \'daily\' or \'monthly\'.'). Errors do not categorize as retryable/user-fixable/fatal or guide the LLM on next steps. E.g., when a project_id does not exist in add_task, the tool will silently insert a task with an orphaned foreign key. No error is returned; no guidance is offered.
Opaque identifiers (numeric IDs) force inefficient tool chains. Users say 'Show me tasks for the Web App project' but the tool requires project_id (an integer). The server offers no way to look up a project by name, the agent must either know the ID or call list_projects, extract the ID, and then call view_tasks. This matches pattern:natural-identifiers guidance: 'Users say \'Send a DM to Jack\', not \'Send a DM to U2405687\'.' Accept names as well as IDs to improve usability.
No pagination support on list_projects. Although only two simple tools (view_tasks, list_projects) return multiple rows, list_projects has no limit, offset, or page parameters. If a user has hundreds of projects, all are returned unfiltered, wasting tokens and potentially exhausting context. The tool should accept limit (default 20, max 100) and offset parameters, and return a total_count field.
Tool descriptions lack WHEN guidance. Descriptions should answer: What does it do? When should the LLM call it? What does it return? For example, 'list_projects' description reads 'List all projects', no guidance on when to use it (e.g., 'Call this first to discover available projects or to map project names to IDs for downstream tool calls'). This forces LLMs to guess when to invoke discovery tools.
No input validation on date format in add_task. The due_date parameter claims format 'YYYY-MM-DD' in the description, but the code does not validate the format before inserting into the database. If an LLM passes '12/25/2024' or 'next week', the database insert will fail with a cryptic error. Add explicit validation with a clear error message and regex pattern in the parameter description.
update_task status parameter lacks enum constraint. The description says status must be 'pending, in-progress, done' but the parameter schema does not declare an enum. LLMs frequently hallucinate values like 'completed', 'closed', 'pending-review', etc. Declare status as enum in the JSON Schema, not just in text description.
No idempotency guarantees or deduplication. If an agent retries add_project with the same name and description, two identical projects are created. For a tool modifying state, idempotency or deduplication is essential to prevent duplicate records on agent retry. Consider accepting an optional idempotency_key parameter or checking for existing projects by name.