Loading ToolBench
An MCP server for managing a product catalog with database persistence using FastAPI and SQLAlchemy with MySQL backend
Product Catalog MCP Server has 2 tools with basic definitions but significant quality gaps. Both tools have acceptable names starting with action verbs (list_, get_), but descriptions lack depth, parameter documentation is minimal, output schemas are not formally documented, and error handling is inadequate for production use. The server uses fastmcp with STDIO transport only. Tool definitions are explicitly registered via @mcp.tool() decorators in mcp_server.py, so they are verifiable, not inferred.
Retrieve details of a specific product by its ID. Args: product_id: The unique identifier of the product
List all available products with their ID, name, price, and description.
No output schemas documented. Both tools return Dict but LLMs cannot know what fields to expect. list_products returns an array of product objects with {id, name, price, description}; get_product returns {id, name, price, description} or {error} on failure. Without formal schema documentation, agents cannot plan downstream tool chains or extract required fields reliably.
list_products lacks pagination. No limit, offset, or cursor parameters. Returns entire product catalog in a single response with no mention of result count limits. If catalog grows to thousands of products, response will bloat context window and degrade LLM reasoning. Baseline pattern requires page/offset/limit + total count or next_cursor.
No parameter descriptions beyond 'The unique identifier of the product' for get_product.product_id. Missing context on valid ID ranges, format constraints, whether IDs are sequential/UUIDs, or how they map to database records. Parameter description is minimal (36 chars) but technically present.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 40 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Error handling is silent for failure cases. get_product returns {"error": "Product not found"} as a dict rather than a structured error with recovery guidance. Per pattern:recovery-guide, error responses must tell the LLM what to do next (e.g., 'Product not found. Call list_products() to see available IDs.'). Returning error as a success response prevents the LLM from recognizing failure and adjusting strategy.
Tool descriptions are generic and lack WHEN/WHY context. list_products says 'List all available products with their ID, name, price, and description' (100 chars), adequate length but no guidance on when to call it or what it reveals. get_product description (97 chars) explains WHAT but not WHY an agent would pick this over list_products + search logic. Descriptions should clarify the distinction and use case.
No input validation or constraints. get_product.product_id is typed as integer but no validation of min/max bounds, whether negative IDs are allowed, or what happens if the ID is unreasonably large (e.g., 99999999999). Baseline pattern:constrained-input requires explicit range or enum constraints to prevent LLM hallucination of invalid values.