Loading ToolBench
An MCP server that enables AI chatbots to interact with Shopify storefronts, manage shopping carts, search products via vector database, and retrieve order information. Integrates FastAPI, OpenAI embeddings, FAISS vector search, and Shopify GraphQL/REST APIs.
This server has foundational tool definitions but suffers from several critical gaps that prevent production deployment. While 8 of 9 tools have reasonable descriptions (50-150 chars on average), the schemas are incomplete or missing in multiple cases. Tool #9 (file_search) has NO input schema visible in the code. Naming is generally clear but could be more specific (e.g., 'get_products_data' vs. 'search_products_by_similarity'). Parameters lack comprehensive type constraints (no enums for variant selection, no numeric bounds for top_k_result). Error handling and output schemas are not documented. This server reads like an OpenAI Tools wrapper rather than a fully-specified MCP interface. Average per-tool score: 42.
Add one or more line items to an existing shopping cart.
Create a new shopping cart with initial items.
File search tool for vector store retrieval using OpenAI's file search feature.
Retrieve and format Shopify order details using an order number.
Fetch the complete and up-to-date product details directly from Shopify using the product's handle.
Get product data for a given query using vector similarity search in the product database.
Tool #9 (file_search) has NO visible input schema and NO description in the tool definition. This violates the critical requirement that every tool must have a description and documented input parameters. The FileSearchToolParam only references vector_store_ids, leaving the LLM unable to understand how to invoke it.
Numeric parameter 'top_k_result' in get_products_data lacks constraints (min/max bounds). An LLM could pass 10000 or negative values, potentially causing API errors or timeouts. Should specify: minimum 1, maximum 100 (or domain-appropriate limit).
Cart operations (create_new_cart_with_items, add_cartline_items, update_cartline_items, remove_cartline_items) accept 'variant' as a free-form string. No enum constraint or format guidance. An LLM cannot know which variant values are valid, should either provide an enum or reference a discovery tool (e.g., get_product_via_handle returns valid variants).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 48 | - | v1 |
Retrieve the current state of a shopping cart.
Remove one or more line items from a shopping cart.
Update one or more line items in a shopping cart (e.g., adjust quantity or variant).
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract relevant fields from responses. For example, does get_product_via_handle return pricing, inventory, images? Does create_new_cart_with_items return a cart_id that can be passed to query_cart? Response structures must be explicitly defined.
Naming ambiguity: 'get_products_data' is generic and does not indicate that this is a vector similarity search. Compare with alternatives like 'search_products_by_similarity', 'find_similar_products', or 'search_products_vector'. The current name does not signal to the LLM when to prefer this over get_product_via_handle.
No error handling or recovery guidance. What happens if a product handle is invalid? If an order number doesn't exist? If a variant is not found in a product? Tools should return structured errors with actionable guidance: 'Product handle "invalid-product" not found. Try search_products_data() to find similar items.'
Destructive operations (create_new_cart_with_items, add_cartline_items, update_cartline_items, remove_cartline_items) have no dry-run, confirmation, or compensation tooling. An LLM could accidentally remove all items from a cart with no undo path. Consider adding a 'confirm_changes' parameter or separate confirmation step.
Cart operations require 'session_id' or 'cart_id' as a string identifier, but there is no discovery tool to list or create carts without items, nor guidance on what valid cart IDs look like. An LLM cannot know how to obtain a valid cart_id except by calling create_new_cart_with_items first. Add a 'get_default_cart' or 'list_carts' tool.
Parameter 'order_number' in get_order_via_order_number accepts both '#1234' and '1234' formats but the description does not explain parsing. Will the tool auto-strip '#'? Will it match both formats? This undocumented flexibility creates ambiguity, either enforce one format or explicitly state the parsing behavior.