Loading ToolBench
MCP server for Shopify Admin API - search orders, check inventory, lookup customers, and get sales summaries
Four tools with clear verb-noun naming (search_orders, check_inventory, lookup_customer, get_sales_summary). All have descriptions (194 - 250 chars, within baseline 34 - 392 range). Input schemas present with Zod validation and type definitions. However, output schemas are not documented in the visible code, responses are inferred from implementation. Parameter descriptions are adequate but lack format constraints (e.g., ISO 8601 date format stated in description but not enforced via regex/pattern). Error handling returns structured responses with isError flag, but recovery guidance is minimal. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all being read-only operations.
Look up current inventory levels for a product by its title or numeric product ID. Use this to answer questions like 'is the Blue T-Shirt in stock?', 'how many units of product 12345 are left?', or 'what SKUs does this product have?'
Generate a revenue summary for a specific date range, including total revenue, number of orders, average order value, and top 5 best-selling products. Use this for questions like 'what were our sales last month?', 'revenue this quarter?', or 'what are our top selling products?'
Search for Shopify customers by email, name, or phone number. Use this to answer questions like 'find customer [email protected]', 'look up customers named Smith', or 'who is the customer with phone 555-1234?'
Search and retrieve Shopify orders filtered by status, date range, or customer email. Use this to answer questions like 'show me recent open orders', 'what orders did [email protected] place?', or 'how many orders were placed last week?'
Output schemas not documented. Tool descriptions mention what is returned (e.g., 'revenue summary, number of orders, average order value, top 5 products') but the actual response structure is not formally specified. LLMs cannot plan downstream tool calls or extract fields reliably without documented output schemas.
Date parameters lack format validation. search_orders accepts 'created_at_min' and 'created_at_max' with description 'ISO 8601, e.g. 2024-01-01' but no regex pattern or format constraint in schema. LLMs may pass invalid formats (e.g., '01/01/2024' or 'Jan 1, 2024'), causing silent API failures.
No tool annotations. All four tools are read-only (no side effects), but readOnlyHint is not set. This prevents clients from optimizing caching, retry logic, or safety checks. Agents cannot distinguish safe-to-retry tools from destructive ones.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Error messages lack recovery guidance. check_inventory returns 'No products found matching "<title>"' but does not suggest next steps (e.g., 'Try a partial title or use product_id if known'). lookup_customer has no documented error cases for ambiguous matches or no results.
lookup_customer accepts a single 'query' parameter (email, name, or phone) but does not document how to disambiguate results if multiple customers match. No pagination or limit parameter visible. Large result sets could exhaust context.