Loading ToolBench
Transactional reliability layer and proxy for Model Context Protocol (MCP) tool execution
AgentX demo server has basic tool definitions with schemas and descriptions, but significant quality gaps. All 4 tools have descriptions (10-50 chars, below the 194-char baseline) and input schemas with typed properties. However, parameter descriptions are minimal or absent (e.g., 'appointmentId' has no description in get_appointment; 'cardNumber' and 'cvv' in process_payment lack descriptions). No output schemas documented. Error handling is minimal, tools return JSON strings without actionable recovery guidance. Security concern: process_payment accepts credit card numbers and CVV as optional parameters, violating secret-injection pattern. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels (READ_ONLY, WRITE, DESTRUCTIVE). Naming is verb-first and clear, but parameter naming lacks type suffixes (e.g., 'patientId' vs 'patient_id').
Book a medical appointment slot for a patient
Cancel an existing medical appointment
Fetch appointment details by patient ID or appointment ID
Charge customer credit card for an order
Credit card and CVV exposed as optional tool parameters (process_payment). Credentials must never appear in tool parameters; use server-side secret injection via environment variables or vault. Agent traces log all parameters, secrets leak into logs and prompt history.
Parameter descriptions missing or trivial. get_appointment properties (appointmentId, patientId, date) have no descriptions. process_payment lacks descriptions for cardNumber and cvv. LLMs cannot infer parameter meaning from names alone, a 'date' param could mean appointment date, birth date, or filter date.
No output schemas documented. Tools return JSON strings without declaring expected fields, types, or structure. LLMs cannot plan downstream calls or extract data reliably. E.g., book_appointment returns appointmentId, status, message, but this is not formally declared.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | <=2025-11-25 | v2 |
No tool annotations despite clear risk levels. Tools marked DESTRUCTIVE (cancel_appointment) and WRITE (book_appointment, process_payment) lack readOnlyHint, destructiveHint, or idempotentHint annotations. Agents cannot distinguish safe-to-retry operations from irreversible ones.
Error handling lacks recovery guidance. Tools return generic JSON error objects (e.g., 'Appointment not found') without actionable next steps. Errors should tell the LLM what to do: 'Appointment not found. Try search_appointments() with patient ID or date range.'