Loading ToolBench
Write PLC ladder logic with just a numpad. A minimal DSL that compiles to IEC 61131-3.
Four tools with complete input schemas and basic descriptions, but significant gaps in parameter documentation and output schema clarity. All tools are read-only domain-specific compilers/validators for BNS Lang → IEC 61131-3. Naming follows verb_noun pattern (bns_compile, bns_check, bns_ladder, bns_explain). However, parameter descriptions are minimal (1-3 words), output schemas are undocumented, and error handling is generic. No tool annotations (readOnlyHint, idempotentHint). Descriptions average ~60 chars, below the 194-char baseline for A+ tools. The 'target' enum in bns_compile is well-constrained, but other parameters lack format guidance.
Validate BNS Lang syntax. Returns OK or errors.
Compile BNS Lang to IEC 61131-3 Structured Text.
Explain BNS Lang code in plain language.
Generate ASCII art ladder diagram from BNS Lang.
Output schemas undocumented. All four tools return { content: [{ type: 'text', text: string }], isError?: boolean } but this structure is not declared in tool definitions. LLMs cannot plan downstream operations or validate responses.
Parameter descriptions are minimal (1-3 words: 'BNS Lang source code', 'BNS Lang source'). Baseline for A+ tools is 72 chars per param. No guidance on expected format, constraints, or when to use each tool instead of others.
No tool annotations. All four tools are read-only and idempotent, but lack readOnlyHint and idempotentHint in schema. LLMs cannot infer safety properties without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
Error handling is generic. When bns.check() or bns.compile() throws, the catch block returns 'Error: {err.message}' with isError: true. No guidance on what went wrong, why, or how to fix it. LLMs cannot self-correct.
Tool descriptions do not explain when to use each tool. bns_compile, bns_ladder, and bns_explain all accept 'code' and produce different outputs, but descriptions do not clarify the use case for each (e.g., 'Use bns_ladder to visualize logic flow; use bns_explain for natural-language summary').