Loading ToolBench
AI-powered multi-agent system for sales consultation and order processing using FastMCP. Provides tools for product inventory checking, order creation, and retrieval via MongoDB.
Server has 3 tools with basic definitions but significant quality gaps. Tool names follow verb_noun convention (create_order, get_order, get_product_info) which is good. However, descriptions are present but vary in quality, parameter descriptions are inconsistent, and output schemas are not formally documented. Error handling exists but lacks recovery guidance. The schema for create_order requires a nested dict parameter which is unusual and poorly documented. No enum constraints on valid inputs. The server depends on MongoDB being pre-configured, creating a deployment risk not mentioned in tool descriptions.
Saves the given order data (dictionary) to a file in the 'orders' subdirectory, with a standardized format. Returns a success message with the filename or an error message.
Retrieves the content of an order file by its order_id. Returns a dictionary with file_content or error message.
Retrieves inventory details from storage based on the product. Input is a JSON string or object with product name, and optionally storage and color.
Output schemas are not formally documented. For create_order, function returns a string message. For get_order, returns a dict with 'file_content' or 'error' + 'status' fields. For get_product_info, returns JSON string with 'status', 'products', or 'error' + 'status'. LLMs cannot plan downstream operations without knowing what fields to expect.
create_order parameter 'order_details' is typed as dict with no schema constraints. The description mentions 'required fields: product, color, storage, quantity, total_price, customer_info' but these are not enforced in the schema, only checked at runtime. No enum for product names, colors, or storage capacities. LLMs cannot know valid values without calling the tool.
get_product_info description is vague: 'Retrieves inventory details from storage based on the product.' Does 'storage' mean stock quantity or physical location? What happens if no matches are found? The function returns JSON string instead of structured object, forcing LLM to parse JSON within the response.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 0 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 39 | - | v1 |
Error handling lacks recovery guidance. create_order returns 'Error: Missing required fields: X, Y, Z' (good) but get_order returns {'error': '...', 'status': 404} with no hint about what to do. get_product_info returns 'No product found matching...' but does not suggest how to broaden the search or list available products.
Inconsistent response types. create_order returns string, get_order returns dict, get_product_info returns JSON string. This forces LLMs to adapt parsing logic per tool and wastes tokens on type conversions.
MongoDB dependency is implicit and mandatory. The server exits with logger.critical() if MongoDB is not reachable, but tool descriptions do not mention this prerequisite. An LLM calling get_product_info without knowing MongoDB must be running will receive a generic 'Cannot connect to MongoDB database' error with no recovery path.
get_order implementation searches for files matching 'order_<order_id>*' pattern but parameter description says 'The order ID to retrieve' without explaining the ID format. Is the full UUID required or just the first 16 hex chars? This ambiguity forces trial-and-error by LLMs.
get_product_info returns bare JSON strings instead of structured objects. Caller must json.loads() the response. LLMs struggle with JSON-within-JSON and waste tokens parsing. Should return dict directly and let MCP serialization handle JSON encoding.
No pagination support. If a product search returns hundreds of variants (e.g., iPhone 15 in 10 colors × 4 storages × 3 variants), the response could bloat and exceed context limits. No limit, offset, or cursor parameters documented.
create_order accepts 'order_details' dict but also checks 'if "order_details" in input_data' and unpacks it, suggesting input might be doubly nested. This defensive logic hints at ambiguous interface design and increased potential for LLM errors.