Loading ToolBench
A Next.js-based customer service AI agent platform with MCP server integration for Shopify and other external services. Provides tool management, agent creation, and integration capabilities.
This MCP server has foundational issues across naming, descriptions, and schemas. While tool names follow verb-noun conventions (searchProducts, getProductDetails, saveCustomerMemory), descriptions are generic and lack the specificity LLMs need for proper tool selection. Input schemas are visible but lack parameter descriptions in several cases, violating the critical requirement that every parameter must have a description. The server conflates tool categories (Shopify product search vs. internal vector memory) without clear delineation in descriptions. Error handling is undocumented. Output schemas are not documented in the provided code. The codebase shows React/Next.js UI components managing tools at a UI level, but the actual MCP tool implementations (lib/mcp/servers/shopify/tools, lib/services/vector.server.ts) are not fully visible, making definitive schema validation impossible for tools 2-5.
Retrieve stored customer memories and context
Get detailed information about a specific product from Shopify
Save customer context and preferences to memory database
Search for products in Shopify store
Search for similar customer memories using vector similarity
Tool descriptions are generic and lack LLM-optimization guidance. Example: 'Save customer context and preferences to memory database' (61 chars) does not explain WHEN to use this vs. alternatives, WHAT happens to data, or what success looks like. No dependency hints or recovery guidance.
Input schema for saveCustomerMemory includes 'metadata' as type 'object' with description 'Additional metadata' but provides no schema for the object properties. LLMs cannot determine what fields are valid in metadata.
searchProducts 'limit' parameter lacks minimum/maximum constraints. Description says '1-250' but this is not enforced in schema as minValue/maxValue constraints. LLMs may pass invalid values.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 37 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 23 | 1.15.1+ | v1 |
No output schemas are documented for any tool. The tool definitions show inputs but the code does not include return type documentation. LLMs cannot plan downstream tool calls without knowing what fields to expect.
getProductDetails, getCustomerMemories, and searchSimilarMemories accept IDs (productId, customerId, organizationId) but do not explain whether these are system IDs only or if human-readable names are accepted. No guidance on resolution strategy.
Memory tools (saveCustomerMemory, getCustomerMemories, searchSimilarMemories) require both customerId AND organizationId as required parameters. No explanation of how an agent discovers or knows these IDs in a typical chat flow. This creates friction in agent workflows.
No error handling guidance provided. Tools lack recovery instructions for common failures (product not found, unauthorized access, vector DB timeout, invalid memory type). LLMs receive raw errors with no actionable next steps.
getCustomerMemories includes optional 'memoryType' filter parameter but the saveCustomerMemory description mentions 'Type of memory: preference, context, or fact', these enum values are not surfaced in getCustomerMemories schema or description.
searchProducts returns results up to 250 items, but no pagination support is documented (no offset/limit, no next_cursor, no total count). Large result sets will blow context windows without pagination guidance.
saveCustomerMemory is a write operation but the tool description does not state that it modifies state or warn of side effects. Agents need explicit markers to understand which calls are idempotent vs. destructive.