Loading ToolBench
Shopify MCP Server - Python implementation for accessing Shopify APIs, documentation, and GraphQL schema validation
Server has basic tool definitions with schemas and descriptions, but falls short of production quality. Tool names follow verb_noun conventions (search_docs, fetch_full_docs, validate_graphql_codeblocks). All 5 tools have descriptions and input schemas are visible. However, descriptions are generic and lack context about when to use each tool vs alternatives. Critical issue: all tools require a 'conversationId' parameter that must come from learn_shopify_api, creating a mandatory dependency that is NOT enforced at the framework level, this is a composition/dependency design smell. Output schemas are not documented (tools return generic dict with 'content' and 'isError' fields, but no structure for the actual response data). Error handling returns text responses rather than categorized, actionable error guidance. Parameters lack depth, descriptions are present but many are single-clause and don't explain validation rules, formats, or dependencies. No tool annotations (readOnly hints, idempotent markers). The conversationId coupling across all tools is a major architectural red flag, it should be handled via resource context or implicit session state, not repeated in every parameter.
Retrieve complete documentation for specific paths from shopify.dev. Provides full context without chunking loss.
Explore and search Shopify GraphQL schemas to find specific types, queries, and mutations. Only use this for the Shopify Admin API.
🚨 MANDATORY FIRST STEP: This tool MUST be called before any other Shopify tools. Teaches about supported Shopify APIs and generates a conversationId that is REQUIRED for all subsequent tool calls.
Search across all shopify.dev documentation to find relevant chunks matching your query. Useful for getting content from many different documentation categories.
Validate GraphQL code blocks against the Shopify GraphQL schema. Ensures they don't contain hallucinated fields or operations.
Mandatory conversationId parameter repeated across all tools creates fragile composition. All 5 tools require conversationId from learn_shopify_api as a prerequisite, but this dependency is not enforced by the framework, agents must remember to extract and pass it, or all subsequent tools fail. This violates the tool-chaining principle and should be handled via implicit context (resource metadata, environment, or per-request session state).
Output schemas are not documented. All tools return dict with 'content' (list of dicts with 'type' and 'text') and 'isError' boolean, but this structure is inferred from code, not documented in tool descriptions or return type annotations. LLMs cannot plan downstream operations without knowing the response structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 45 | - | v1 |
Tool descriptions lack 'when to use' guidance. Descriptions state WHAT each tool does but not WHEN to select it vs similar tools. For example, search_docs_chunks vs fetch_full_docs both retrieve documentation, when should an agent choose one over the other? This forces LLMs to guess, increasing selection errors.
Error responses are free-text strings without categorization or recovery guidance. When a tool fails (e.g., 'Error fetching getting started information for...'), the response is unstructured text. LLMs cannot determine if the error is retryable, user-fixable, or fatal, they cannot self-correct intelligently.
Parameter descriptions are minimal and lack constraint details. 'prompt': 'The search query for Shopify documentation' does not specify length limits, format requirements, or examples. 'max_num_results': 'Maximum number of results to return' lacks bounds (minimum, maximum). LLMs cannot infer valid ranges and may pass invalid values.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All tools are marked as READ_ONLY in documentation, but this is not enforced in the tool definitions via annotations. Agents cannot reliably determine which tools are safe to retry or compose without side effects.