Loading ToolBench
FastAPI-based MCP server with OpenAI Agents integration for Meridian Electronics product catalog queries, search, pricing, and company information. Supports optional external MCP servers via stdio transport.
Four tools with basic descriptions and partial schemas. All tools have descriptions (10-200 chars range) and input parameters with types, but lack output schema documentation. Parameter descriptions are present but minimal. No error handling guidance, no enum constraints, no pagination support for search results. Tools follow verb_noun naming (fetch_, search_, product_, company_) but descriptions are terse and lack WHEN/WHY context. No tool annotations (readOnlyHint, etc.). Output is JSON strings, not structured objects with documented fields.
Retrieve company information and facts about Meridian Electronics from the catalog.
Fetch a product by lookup (SKU, ME- id, or natural name). Returns product details if found, or not_in_catalog if not found.
Get pricing and stock information for a product by product_id. Returns price_usd, currency, and in_stock status.
Search products by query string with optional limit. Returns matching products with id, sku, name, category, price, currency, and in_stock status.
No output schema documentation. Tools return JSON strings with undocumented fields (found, reason, product, price, count, products, none_match). LLMs cannot plan downstream calls or extract data reliably without knowing response structure.
Descriptions lack WHEN/WHY context and are under 100 chars. 'Fetch a product by lookup' does not explain when to use fetch_product vs search_products. 'Get pricing and stock information' is generic. LLMs need explicit guidance on tool selection.
search_products returns up to 50 items but no pagination support (no offset, cursor, or next_token). Large result sets waste context tokens and degrade LLM reasoning. No limit enforcement in description.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
No error handling guidance. Tools return JSON with 'found: false' and 'reason' fields, but descriptions do not explain what LLM should do next (retry, ask user, try alternative tool). No recovery hints.
No tool annotations. Tools are read-only but lack readOnlyHint metadata. This prevents clients from optimizing caching, retry logic, or safety checks.