Loading ToolBench
OpsPilot has 7 tools with basic schemas and descriptions, but significant quality gaps. Tool definitions are split across multiple files (mcp_server.py, evals/dev_cases.py, agents/graph.py), making registration unclear. Only 2 tools (get_order, get_tracking_status_mcp) are explicitly visible in mcp_server.py with @mcp.tool() decorators; the other 5 are inferred from test/agent code, capping their scores at 50. Descriptions are present but minimal (10-50 chars), below the 50-200 char baseline for LLM-optimized tools. Parameter descriptions exist but are generic ('The order identifier'). No output schemas documented. Error handling is absent, no recovery guidance, no categorization of retryable vs fatal errors. No tool annotations (readOnlyHint, destructiveHint). Naming is verb-first and clear, but duplicate tools (get_tracking_status vs get_tracking_status_mcp) signal poor composition.
Cancel a specific order.
Check whether a specific order is eligible for a refund.
Escalate an order case for human review.
Retrieve details for a specific order.
Retrieve the current tracking status for a specific order.
Retrieve the current tracking status for a specific order.
Issue a refund for a specific order.
Tool registration unclear, only 2 tools explicitly decorated with @mcp.tool() in mcp_server.py; 5 tools inferred from test/agent code without visible MCP registration. Cannot verify if all 7 are actually exposed via MCP protocol.
Duplicate tools: get_tracking_status and get_tracking_status_mcp both retrieve tracking status with identical schemas. LLMs will waste reasoning cycles deciding between them. Consolidate to one canonical name.
Descriptions are 10-50 characters, well below the 50-200 char baseline for LLM-optimized tools. E.g., 'Retrieve details for a specific order.' lacks context on when to use it vs other tools, what fields are returned, or prerequisites.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | <=2025-11-25 | v2 |
No output schemas documented. LLMs cannot plan downstream calls or extract required fields. E.g., does get_order return customer_id, created_at, items? Does cancel_order return a confirmation ID for audit trails?
No error handling guidance. Destructive tools (cancel_order, issue_refund, escalate_case) lack recovery hints, retryability classification, or confirmation steps. An LLM cannot distinguish between 'order not found' (user-fixable) and 'database timeout' (retryable).