Loading ToolBench
Single tool with minimal quality. The tool 'fetch-products' has a basic input schema (Zod-validated string parameter), but critical areas are severely deficient: parameter lacks meaningful description beyond 'the product to fetch'; no output schema documented; error handling is basic and non-actionable; no tool annotations despite being read-only. The description is extremely generic (10 words) and does not explain WHEN to use this tool, WHAT data structure to expect, or HOW to interpret results. Token usage is high due to unstructured JSON responses. No pagination support despite fetching 'first: 10' results. No rate limiting or timeout guidance. Credentials hardcoded as empty placeholders in the source rather than injected from environment. This is a bare-minimum proof-of-concept, not production-grade.
Fetch products from a given shopify store
Parameter description is trivial ('the product to fetch'). Does not explain format, constraints, or valid values. LLM has no guidance on whether to pass a product name, SKU, category, or search query.
No output schema documented. Tool returns raw JSON without specifying field names, types, or structure. LLM cannot reliably extract or chain results.
Tool description is 8 words. Rubric baseline for good descriptions is 50-200 chars, this is 35 chars. Does not explain WHEN to use it vs other product lookup methods, WHAT the response contains, or any prerequisites.
No error guidance. Error responses return raw JSON.stringify(errors) or generic exception messages. LLM cannot determine if the failure is retryable, user-correctable, or fatal.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 32 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 38 | - | v1 |
Credentials (SHOPIFY_STORE and SHOPIFY_ACCESS_TOKEN) are empty strings in the source code. Should be injected from environment variables or a secure vault, never hardcoded.
No tool annotation despite being read-only. Tool definition should include readOnlyHint: true to signal to the agent that no side effects occur.
Response is unstructured JSON dump. 'Found X products: [full JSON]' wastes tokens on GraphQL response nesting. Should strip to essentials (id, title, price, handle, variants simplified) and implement pagination with limit and cursor.
No pagination support. Tool hardcodes 'first: 10' in the GraphQL query but does not expose limit/offset parameters to the agent. Large result sets will blow context windows.
No timeout declared for fetch() call. External API can hang indefinitely, blocking the entire agent.