A customer support chatbot that integrates with an MCP server to provide product information, order management, and customer authentication. Uses LLM-based intent detection to route customer queries to appropriate tools.
This server has significant definition quality gaps. Tool names follow verb_noun convention (good), but descriptions are minimal (10-50 chars, well below the 194-char baseline). Most critically, parameter descriptions are missing or trivial across all tools. Input schemas are present but lack depth, no enums for constrained values (e.g., order status), no format specifications, no validation guidance. Output schemas are not documented. Error handling is absent, no recovery guidance, no categorization of retryable vs fatal errors. The create_order tool (WRITE operation) lacks confirmation/dry-run support. No tool annotations (readOnlyHint, destructiveHint) despite clear risk differentiation.
Creates a new order for a customer with specified items
Retrieves detailed information about a specific order
Retrieves detailed information about a specific product by SKU
Lists orders for a customer, optionally filtered by status
Lists all products, optionally filtered by category and active status
Searches for products matching a query string
Verifies a customer's identity using email and PIN credentials
Parameter descriptions missing or trivial. All 7 tools have parameters with minimal guidance (e.g., 'Customer ID to retrieve orders for' lacks format, constraints, or examples). LLMs cannot infer whether to pass opaque IDs, emails, or names.
No output schemas documented. Callers cannot plan downstream tool chains or extract required fields. E.g., does list_products return product_id, sku, or both? Does get_order return customer_id for chaining to list_orders?
create_order (WRITE operation) lacks confirmation/dry-run support and error recovery guidance. No indication of what happens on partial failure (e.g., 1 of 3 items fails). Agents cannot safely retry.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 42 | 2025-06-18+ | v2 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Despite clear risk differentiation (6 READ_ONLY, 1 WRITE), tools lack machine-readable hints for agent safety policies.
No enum constraints for status filters (list_orders accepts 'status' string with no valid values documented). LLMs will hallucinate invalid statuses; no validation guidance provided.