Loading ToolBench
Create ltspice circuits with LLMs. Convert netlist to LTSpice .asc.
Critical gaps across all dimensions. Tool descriptions are present but minimal (10-20 chars). The `execute` tool has a vague description ('Execute one PyLTSpice API request') that does not explain WHEN to use it or what it returns. Input schemas exist for `execute` but are incomplete: `api_name` and `inputs` lack type constraints, enums, or validation guidance. Three tools (`runtime_info`, `execute_status`, `stop_reset`) have ZERO input parameters documented, they appear to accept no inputs, yet their schemas are not explicitly shown. No output schemas are documented for any tool. No error handling guidance. No parameter descriptions beyond names. Naming is acceptable (verb-noun pattern) but lacks clarity on tool composition, `execute` is dangerously generic and could be split into domain-specific operations.
Execute one PyLTSpice API request asynchronously.
Poll status of last queued/active LTspice operation.
Check LTspice runtime configuration and host diagnostics.
Stop/reset queued operations and clear server object state.
No output schemas documented. LLMs cannot plan downstream operations or extract required fields. Every tool response is opaque.
Input schemas for `execute` lack type constraints and validation guidance. `api_name` is a free-form string with no enum or pattern. `inputs` is an untyped object. No description of valid API names or input structure.
Tool descriptions are too brief (10 - 20 chars) and lack context. 'Check LTspice runtime configuration' does not explain WHEN to call it, what it returns, or how it differs from `execute_status`. No recovery guidance on errors.
`execute` tool is dangerously generic. It accepts arbitrary PyLTSpice API names and inputs, making it impossible for LLMs to validate calls or understand side effects. No indication of which APIs are read-only vs. destructive.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | <=2025-11-25 | v2 |
No error handling guidance. Tools return `dict` responses with no documented error format, recovery steps, or actionable messages. LLMs cannot distinguish retryable failures from user errors.