Loading ToolBench
Two tools with basic definitions but significant gaps. ask_kiro has a short description (79 chars, below baseline 194) and a single string parameter with minimal context. get_kiro_quota has a reasonable description (95 chars) but returns a dict with no documented schema. Neither tool has parameter descriptions beyond the implicit type. No error handling guidance, no output schema documentation, and no tool annotations. The server wraps an external CLI (kiro-cli) without exposing its complexity or constraints to the LLM.
Ask a technical/IT question to Kiro. Returns a markdown-formatted answer.
Check current Kiro quota usage. Returns pct used, overages status, and reset date.
ask_kiro parameter 'question' lacks description. LLM cannot infer scope, format, or constraints (e.g., max length, expected domain, language).
get_kiro_quota returns a dict with undocumented schema. LLM cannot plan downstream usage or extract fields reliably. Should document: {'pct': int, 'overages': bool, 'resets_on': str}.
ask_kiro quota check raises RuntimeError with a message, but no guidance on recovery. LLM sees 'quota exhausted' but doesn't know to call get_kiro_quota first or wait for reset.
ask_kiro subprocess call has a 300-second timeout but no timeout error handling or user-facing message. If kiro-cli hangs, LLM receives a bare exception.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
No tool annotations (readOnlyHint, idempotentHint). Both tools are read-only and idempotent, but LLM cannot infer this from definitions.