Loading ToolBench
Get cloud cost summary for multiple vendors and months
Single tool 'get-cost' has a clear action verb and basic schema, but suffers from critical gaps: (1) description is only 56 chars, below the 194-char baseline and lacking WHEN/WHY context; (2) parameters lack descriptions entirely, 'vendors' and 'months' have no guidance on format, constraints, or examples; (3) no output schema documented, LLMs cannot predict response structure; (4) no error handling guidance, what happens if cost.json is missing or malformed?; (5) no pagination despite potentially returning large datasets. Tool is functional but falls well short of production quality.
Get cloud cost summary for multiple vendors and months
Parameter descriptions missing entirely. 'vendors' and 'months' have no guidance on format, valid values, or constraints. LLMs cannot infer whether to pass ['AWS', 'azure'] or ['aws', 'AZURE'] or full names.
Tool description is only 56 characters. Lacks WHEN to use it, WHAT it returns, and any prerequisites. Baseline is 194 chars; descriptions under 20 chars score 0-20.
No output schema documented. Response is free-form text; LLMs cannot predict structure or extract fields for downstream tool calls. Should return structured JSON with vendor, month, cost, accountId, productName, regionName.
No pagination support. Tool reads entire cost.json and returns all matching records as unstructured text. Large datasets will blow context window. Should accept limit and offset/cursor parameters.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Error handling is silent and unhelpful. If cost.json is missing, returns 'Failed to load cost data' with no recovery guidance. Should tell LLM: 'Cost data unavailable. Check server configuration or try again later.'