Loading ToolBench
A workshop repository containing multiple MCP server implementations for retail use cases and AWS cloud deployment, including order management, product information, and a Streamlit-based MCP playground client.
This is a workshop/educational repository with 11 tools spread across multiple labs. Critical issues: (1) Duplicate tool definitions (create-order, check-order-status, get-product, search-products appear 2-4 times with inconsistent schemas); (2) Most descriptions are extremely brief (10-30 chars), below the 50-200 char LLM-optimized baseline; (3) Input schemas present but minimal, many lack proper type constraints, enums, or ranges; (4) No output schemas documented; (5) No error handling guidance; (6) Tool names lack clarity on when to use one variant over another. The hello/echo/get-system-info tools are trivial. The retail tools (create-order, search-products) have slightly better structure but still fall short of production quality.
Check the status of an existing order
Check the status of an existing order
Create a new order for products
Create a new order for products
Echo back a message
Get detailed information about a specific product
Get detailed information about a specific product
Get system information (simulated)
Duplicate tool definitions with inconsistent schemas. create-order, check-order-status, get-product, and search-products are defined multiple times across labs with different input parameters (e.g., search-products v1 has query+category; v2 adds maxPrice+inStockOnly). LLMs cannot disambiguate which variant to call.
Descriptions are too brief (10-35 chars). Baseline for LLM-optimized descriptions is 50-200 chars. Current descriptions lack context on WHEN to use the tool, WHAT it returns, and any prerequisites. E.g., 'Say hello to someone' does not explain when an agent should invoke this vs other tools.
No output schemas documented. Tools return data but LLMs have no specification of response structure, field types, or what IDs/references are included for chaining. This forces LLMs to infer output shape and risks broken downstream tool calls.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | <=2025-11-25 | v2 |
Say hello to someone
Search for products by name or category
Search for products by name or category
Input schemas lack constraints. search-products accepts optional query and category but no enums, patterns, or length limits. create-order accepts quantity as number with no min/max bounds, LLMs could pass negative or absurdly large quantities.
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. A failed create-order returns nothing, the agent cannot distinguish 'out of stock' from 'invalid product ID' from 'permission denied'.