Loading ToolBench
MCP server providing GraphQL tools for querying a sales database with order, customer, product, and analytics data
The server has 7 tools with basic functionality covering GraphQL query operations against a sales API. All tools have descriptions and most have input schemas, but quality is inconsistent. Naming follows verb_noun convention (get_, test_), which is good. However, descriptions are minimal (avg 60 chars vs. baseline 194 chars), parameter descriptions are missing entirely in most tools, and output schemas are not documented. Error handling returns structured responses with 'error' and 'success' fields, but lacks actionable recovery guidance. The graphql_query tool is overly generic and allows arbitrary queries without validation, which is a security concern. Parameter descriptions are almost completely absent, only order_id, customer_name, and customer_id receive minimal inline documentation in the source, not in formal schema descriptions.
Retrieve all orders with complete details.
Retrieve a specific order by its ID.
Retrieve all orders for a specific customer by ID.
Retrieve all orders for a specific customer by name.
Get the total amount spent by a customer.
Run a raw GraphQL query against the sales API.
Test the connection to the GraphQL endpoint.
Parameter descriptions completely absent from JSON schemas. Input parameters (e.g., 'query', 'variables', 'customer_name', 'order_id') have inline docstring descriptions in Python but lack descriptions in the MCP tool schema that LLMs actually see. This violates the pattern that every parameter needs a description for LLM selection.
No output schema documented for any tool. LLMs cannot determine what fields to expect in responses (orders vs order, success vs data, error structure). This prevents proper chaining and forces LLMs to infer structure from runtime behavior.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 0 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 40 | - | v1 |
graphql_query tool allows arbitrary GraphQL queries with no validation or sanitization. This is a security risk, a compromised or adversarial LLM could execute mutation queries, delete data, or access unauthorized fields. Consider restricting to query-only (no mutations) or requiring explicit queries.
Descriptions are too short (avg 60 chars vs baseline 194 chars). They state WHAT the tool does but lack WHEN to use it or any prerequisites. Example: 'Retrieve all orders with complete details' doesn't explain when to call this vs get_order_by_id or get_orders_by_customer_name. Descriptions should be 10-200+ chars with decision-making guidance.
Error handling lacks actionable recovery guidance. Responses return {error, success} but don't tell LLMs what to do next. Example: If get_order_by_id fails with 'Order not found', it should suggest 'Try get_all_orders() to list available orders' or offer available alternatives.
No pagination support. get_all_orders has no limit or offset parameters, risking context window exhaustion if there are hundreds or thousands of orders. Even with pagination returning in the response, the tool description doesn't mention that results are limited.
Hardcoded GraphQL endpoint (http://127.0.0.1:5001/graphql) is localhost-only and not configurable. This prevents deployment to production or cloud environments where the endpoint would differ. Should use environment variables or configuration.