Loading ToolBench
MCP server providing tools for boiler maintenance operations, including customer information retrieval, maintenance scheduling, purchase processing, and email notifications
This server has 4 tools with schemas present in test files and source code. However, there are significant quality issues: (1) Tool naming lacks action verbs - 'boiler-info' should be 'get_boiler_info', 'maintenance' is vague (should be 'get_maintenance_schedule' and 'schedule_maintenance'), 'purchase' should be 'create_purchase_order' or 'process_purchase', 'email' should be 'send_email'. (2) Descriptions are present but generic and lack WHEN/WHY guidance. (3) Parameter descriptions exist but are minimal (10-30 chars average) and lack format constraints, ranges, or dependency documentation. (4) The 'purchase' tool exposes sensitive card details (card_number, cvv, expiry_date) as tool parameters, critical security violation. (5) Output schemas are not visible in the source code provided, only test expectations. (6) No evidence of error handling patterns, recovery guidance, or actionable error messages. (7) 'maintenance' tool combines two distinct actions (get, schedule) with conditional required parameters, should be split into two tools per composition pattern.
Get customer boiler information and warranty details
Send confirmation and notification emails to customers
Schedule maintenance services or get maintenance information. Use this when a user wants to schedule boiler maintenance, check maintenance status, or get maintenance recommendations.
Process boiler purchase orders with payment processing and installation scheduling
CRITICAL: 'purchase' tool exposes sensitive payment credentials (card_number, cvv, expiry_date) as input parameters. Credentials must never appear in tool params; they must be injected server-side from a vault. This violates pattern:secret-injection and creates security audit risk, all parameters are logged in agent traces.
Tool naming lacks action verbs. 'boiler-info' should be 'get_boiler_info'; 'maintenance' should be split into 'get_maintenance_schedule' + 'schedule_maintenance'; 'purchase' should be 'create_purchase_order'; 'email' should be 'send_email'. LLMs infer tool intent from the name before reading descriptions, generic names cause misselection.
'maintenance' tool violates single-responsibility principle. It combines two distinct actions (GET schedule vs SCHEDULE new service) with conditional required fields (service_date, service_type only needed for 'schedule' action). JSON Schema does not express this conditional logic, and the description does not guide LLM on when to use each action. Split into 'get_maintenance_schedule' (customer_id only) and 'schedule_maintenance' (customer_id, service_date, service_type required; issue_description conditionally required).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 40 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 47 | - | v1 |
All tool descriptions lack WHEN/WHY context. They state WHAT the tool does but not when to use it instead of a similar tool, or what prerequisites exist. E.g., 'maintenance' description says 'Schedule maintenance services or get maintenance information' but does not say when to call GET vs SCHEDULE, or whether the customer must be in the system already.
Output schemas are not documented in source code. Test file references expected fields (e.g., 'customer', 'boiler', 'warranty' for boiler-info; 'order', 'payment', 'boiler', 'installation' for purchase) but no JSON Schema is visible for responses. LLMs cannot plan downstream tool calls without knowing response structure.
Parameter descriptions are too short (average 10-25 chars). Rubric baseline is 72 chars. E.g., 'customer_id' described as 'The customer ID for retrieving boiler information' (54 chars) lacks format guidance. 'issue_description' is '23 chars ('Description of the issue for emergency/repair services') without explaining what level of detail is expected or maximum length.
'purchase' tool parameters lack format/range constraints. 'card_number' has no stated length (16 digits for Visa/MC?). 'expiry_date' says 'Card expiry date' without format (MM/YY? MM/YYYY?). 'cvv' has no length constraint (3-4 digits?). 'amount' is numeric with no min/max bounds. These unmotivated parameters invite hallucinated values from LLMs.
No error handling patterns visible. No evidence of recovery guidance (e.g., 'If customer not found, call search_customers first'), error classification (retryable vs fatal), or actionable error messages. Tools return success/fail in test but no specification of error conditions or how LLM should respond.
No permission gates or scope declarations visible. The 'purchase' tool processes irreversible financial transactions without evidence of permission checks or audit logging. No pattern:permission-gate or pattern:scope-declaration implemented.
'email' tool description lacks clarity on what email_type values mean and when each is used. Enum has 'confirmation', 'reminder', 'alert', 'notification' but descriptions do not explain the distinction or provide guidance on which to choose for a given scenario.