Loading ToolBench
Custom MCP tools for interacting with Shopify Admin API
The Shopify MCP server defines 6 tools with complete input schemas and descriptions. All tools use action verbs (list_*, query_*) following naming conventions. However, descriptions are present but generic (typically 10-50 chars), falling below the 50-200 char ideal for LLM optimization. Parameter schemas are well-typed with defaults, but lack granular descriptions for many fields (e.g., 'verify_ssl' is described only as 'Whether to verify SSL certificates (default: False)' with no guidance on when to use). Output schemas are not documented, responses are inferred as Dict[str, Any] with no field-level specification. Error handling returns basic error dicts but lacks recovery guidance or error categorization. The query_shopify_custom_tool is a powerful escape hatch but lacks controls on dangerous mutations (allow_mutations=false default is good, but no dry-run or confirmation pattern). Read-only tools are marked with risk classification, but no tool-level security annotations (readOnlyHint/destructiveHint) are visible in the implementation.
List Shopify discount nodes using GraphQL.
List Shopify inventory items using GraphQL.
List Shopify marketing activities using GraphQL.
List Shopify orders using GraphQL.
List Shopify products using GraphQL.
Executes custom GraphQL queries or mutations against the Shopify Admin API.
Output schemas not documented. All tools return Dict[str, Any] with no field-level specification. LLMs cannot infer what fields to expect or how to chain results to downstream tools.
Parameter descriptions are generic and lack actionable guidance. Example: 'verify_ssl: Whether to verify SSL certificates (default: False)' does not explain WHEN to use this, which values are safe, or consequences of disabling. Expected: 'Whether to verify Shopify API SSL certificates. Set to false only in development/testing; production use requires true.'
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 47 | - | v1 |
Error responses return generic error dicts {error: ..., message: ...} with no recovery guidance or error classification. Example: catching Exception and returning {error: 'Execution error', message: str(e)} gives LLM no actionable next step. Expected: 'Invalid product status: got "PENDING", must be one of: ACTIVE, DRAFT, ARCHIVED. Try list_products_tool(status="ACTIVE") instead.'
query_shopify_custom_tool lacks mutation controls. While allow_mutations defaults to false, there is no dry-run mode, confirmation pattern, or warnings for irreversible mutations. An LLM could enable mutations and execute destructive queries without safeguards.
Tool-level security annotations missing. No tool declares readOnlyHint, destructiveHint, or idempotentHint. This prevents clients from applying UI/UX restrictions or safety policies based on tool properties.
Tool descriptions are too short (mostly <50 chars). Pattern baseline: 50-200 chars for LLM optimization. Example: 'List Shopify products using GraphQL.' does not explain WHEN to use this vs other discovery tools, what fields are returned, or prerequisites.
Pagination metadata not documented. Tools accept after/before cursor strings, but no documentation of pagination structure, total count, or hasNextPage indicator. LLMs cannot determine when to stop paginating or how many results exist.
Credentials passed via environment variables (SHOPIFY_SHOP_URL, SHOPIFY_ACCESS_TOKEN) are resolved at tool invocation time, not server init. This is correct (secret injection), but resolve_credentials() error handling returns a generic error dict rather than guiding the user to set env vars or pass credentials.