Loading ToolBench
MCP server connecting AI customer support agents to Zendesk, Stripe, and Shopify
Server has 12 tools with complete schemas and descriptions, but quality is uneven. Naming follows verb_noun convention well (get_, search_, update_, add_, process_, cancel_). Descriptions are present (avg ~120 chars) but lack actionable context, many omit WHEN to use the tool or what happens on failure. Parameters are typed and mostly described, but lack constraints (enums, ranges, format hints). Output schemas are not documented, LLMs cannot predict response structure. Error handling is absent from visible code. Security concern: no evidence of secret injection for API credentials (Zendesk, Stripe, Shopify tokens likely hardcoded or in .env without server-side gating). Composition is reasonable, tools are single-purpose and chainable (e.g., search_tickets → update_ticket_status). However, no pagination support visible, and result limits are not enforced or documented.
Look up a Shopify order by its ID. Returns order details including items, pricing, shipping address, and fulfillment status.
Get shipping tracking information for a Shopify order. Returns carrier, tracking number, current status, and estimated delivery date.
Search Shopify orders by customer email address. Returns all orders for the customer.
Cancel a Stripe subscription. Can cancel immediately or at end of billing period.
Look up a Stripe payment by its payment intent ID. Returns amount, status, customer info, and refund eligibility.
Look up a Stripe subscription by ID. Returns plan details, billing period, and cancellation status.
Output schemas not documented. LLMs cannot predict response structure (fields, types, nested objects). Agents cannot plan downstream tool calls or extract required data reliably.
Descriptions lack actionable context. Missing WHEN to use each tool, prerequisites, and error recovery guidance. Avg description length ~120 chars is acceptable, but content is generic (e.g., 'Returns ticket details' without explaining when to call vs. search_tickets).
No pagination or result limits documented. search_tickets, stripe_search_payments, shopify_search_orders could return hundreds of items, blowing context window. No limit, offset, or cursor parameters visible.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | <=2025-11-25 | v2 |
Process a refund for a Stripe payment. Supports full or partial refunds. Checks eligibility before processing.
Search Stripe payments by customer email address. Returns all payments for the customer.
Add a comment to a Zendesk support ticket. Can be public (visible to customer) or internal.
Look up a Zendesk support ticket by its ID. Returns ticket details including subject, status, priority, requester info, and comments.
Search Zendesk tickets by keyword or requester email. Returns matching tickets.
Update the status of a Zendesk support ticket. Valid statuses: new, open, pending, hold, solved, closed.
No error handling guidance in visible code. Agents cannot distinguish retryable errors from user-fixable ones. E.g., 'ticket not found' should suggest search_tickets; 'invalid status' should list valid enums.
Credentials (Zendesk, Stripe, Shopify API keys) likely exposed in environment or hardcoded. No evidence of server-side secret injection or permission gating. Agent traces will log credentials if passed as parameters.