Loading ToolBench
Expose internal systems to LLM agents over the Model Context Protocol.
Toolkit provides 13 tools across file, REST, SQL, and order management domains. All tools have descriptions (10-100 chars) and input schemas with typed parameters. However, descriptions are uniformly terse (avg 50 chars vs baseline 194), lack actionable context for LLM selection, and omit dependency hints. Parameter descriptions are minimal or absent. No output schemas documented. Error handling is generic. Tool naming is inconsistent: 'lookup_order' appears 3 times (duplication), 'add' and 'shout' are generic verbs, 'refuse' is non-standard. No tool annotations (readOnlyHint, destructiveHint). Security patterns present (SQL allowlist, redactor, audit log) but not reflected in tool definitions. Composition is weak: multiple tools do similar things (lookup_order, orders.lookup, search_orders) without clear differentiation.
Add two integers.
List the tables available to this server.
Run a read-only SQL query against orders, order_lines. Returns at most 100 rows.
List readable files, recursively.
List known order identifiers.
Look up the status of an order by its identifier.
Fetch one order by its internal identifier.
Look up one order by its internal identifier.
Duplicate tool names: 'lookup_order' registered 3 times (examples/rest_server.py, examples/quickstart.py, go/mcp/server_test.go). LLMs cannot distinguish between them; only the last registration survives. Consolidate into a single canonical tool or rename variants (e.g., lookup_order_by_id, lookup_order_with_expansion).
Descriptions are uniformly terse (avg 50 chars vs baseline 194). Examples: 'List readable files, recursively.' (33 chars), 'Always fails.' (12 chars). Descriptions lack WHEN to use, prerequisites, or recovery hints. LLMs cannot reliably select tools or plan multi-step sequences.
Parameter descriptions are missing or minimal. Examples: 'subdirectory' in list_files has description but 'expand' in lookup_order is vague ('Comma-separated relations to expand, e.g. "lines".'); 'status' in search_orders lacks enum constraint in description. LLMs cannot infer valid values or formats.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
Read one text file from the exposed directory.
Always fails.
Find lines containing *pattern* across the exposed files.
Search orders by customer email or status.
Uppercase a string. Useful for checking the wiring.
No output schemas documented. Tools return data but LLMs have no formal specification of response structure, field types, or required fields. Agents cannot plan downstream tool calls or extract data reliably.
Generic tool names reduce clarity: 'add' (ambiguous, add what?), 'refuse' (non-standard, should be test_error or fail_intentionally), 'shout' (informal). Naming should follow verb_noun pattern (e.g., uppercase_text). LLMs struggle to infer intent from vague names.