Loading ToolBench
A conversational product discovery service that ranks and filters product records based on user queries using keyword matching and relevance scoring
This server has a single tool with significant structural and design issues. The tool name lacks clarity (no leading verb), parameter descriptions are absent despite schema presence, and the implementation reveals poor abstraction. The tool performs keyword matching on raw dictionary values, a brittle, low-signal ranking approach. Output includes redundant fields ('rank' initialized to 0 then overwritten) and truncated evidence text without explanation. Error handling is minimal: only two validation errors for input types, no recovery guidance. The server implements a custom STDIO protocol with line-delimited JSON, reinventing MCP's message handling rather than following the specification. No support for schemas, descriptions of parameter semantics, or idempotent operation hints. This is a proof-of-concept, not a production-grade tool.
Ranks and filters product records based on a user query using keyword matching. Returns matched records sorted by relevance score.
Tool name lacks action verb. 'conversational_product_discovery' is a noun phrase, not a verb_noun pattern. Should be 'search_products' or 'rank_products_by_query'.
Parameter 'records' description is vague: 'List of product record dictionaries to search through' does not specify expected schema, required fields, or max size. Agents cannot infer if records should have 'id', 'name', 'description', or other fields.
Parameter 'query' description lacks detail. Does not specify minimum/maximum length, supported operators, or behavior with special characters. LLMs cannot determine valid input range.
Output schema not documented. Response includes fields 'status', 'query', 'matches', 'count', and per-match 'rank', 'record_index', 'score', 'evidence', but LLMs have no explicit specification of these fields, their types, or semantics. 'evidence' is truncated text (300 chars) without explaining why.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 22 | 2026-07-28+ | v2 |
Tool description is too generic and omits critical context. 'Ranks and filters product records based on a user query using keyword matching' does not explain when to use this vs. other search/ranking strategies, what 'relevance score' means (term count / total terms?), or how ties are broken.
Error handling is minimal and non-actionable. Only two input validation errors ('records must be a list', 'query must be non-empty') are caught. No recovery hints. A generic catch-all returns {'status': 'error', 'error': str(error)}, the LLM receives a bare exception message with no next steps.
Ranking algorithm is too simplistic and lacks transparency. Scores are computed as (matching_terms / total_terms), which treats all records equally if they match the same number of terms. No ordering for ties; no weighting by field (e.g., 'title' match > 'description' match); no handling of multi-word phrases. Evidence is truncated to 300 chars without explanation, making it hard for LLMs to trust the ranking.
No pagination support. The tool returns all matches without limit. If there are 1000 matching products, all are returned, exploding context and token usage. Should support page/offset and limit parameters, and document max result size.
Input validation is incomplete. The tool checks if 'records' is a list and 'query' is a non-empty string, but does not validate record structure (does not require specific fields), query length bounds, or record count bounds. An adversarial input (10 million records, 1 MB query) could cause DoS.