Loading ToolBench
The Modbus MCP server presents well-structured tool definitions with consistent naming, proper schemas, and tool annotations. All 11 tools follow verb_noun patterns (read_*, write_*). Input schemas are present and typed for all tools. Descriptions are concise and action-oriented (average ~60 chars, within the 10-1024 char guideline). However, parameter descriptions lack depth, they state WHAT (e.g., 'Coil value to write') but rarely state WHEN to use the tool or dependencies. Output schemas are not documented in the visible code, forcing LLMs to infer result structure. Error handling is present (RuntimeError thrown on Modbus failures) but lacks recovery guidance or categorization. The tool set demonstrates good composition (read/write separation, batch variants for coils/registers, specialized operations like mask_write_register), but lacks domain-specific context that would help LLMs choose between similar tools (e.g., when to prefer read_write_registers over separate read/write calls). Tool annotations are correctly applied (readOnlyHint, openWorldHint) and match tool semantics. Baseline comparison: average tool description length 71 chars (p10=34, p90=392) is acceptable; all params have type and description (100% coverage matches A+ baseline), but descriptions are perfunctory rather than LLM-optimized.
Modifies a single register using a bit mask on a remote unit.
Reads one or more coils on a remote unit.
Reads one or more discrete inputs on a remote unit.
Reads exception status from a remote unit.
Reads device information from a remote unit.
Reads the contents of one or more registers on a remote unit.
Reads and writes registers on a remote unit in a single operation.
Output schemas not documented. Tool descriptions state what is returned (e.g., 'coil values') but lack formal schema documentation, forcing LLMs to infer result structure. This violates the pattern:tool-description and pattern:response-shaper baselines where 100% of A+ tools document return types.
Parameter descriptions lack context depth. Descriptions are minimal (e.g., 'Coil value to write', 'Starting coil address') and do not explain WHY an LLM should choose this tool over alternatives, WHEN to use it, or dependencies (e.g., when host/port/unit are required vs optional). This falls short of the pattern:tool-description baseline of LLM-optimized 50 - 200 char descriptions that include WHAT, WHEN, and prerequisites.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 70 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 50 | - | v1 |
Writes data to a single coil on a remote unit.
Writes data to one or more coils on a remote unit.
Writes data to a single register on a remote unit.
Writes data to one or more registers on a remote unit.
Error handling lacks recovery guidance and categorization. Code shows RuntimeError raised on Modbus failures (e.g., 'Could not read X from host:port'), but errors do not categorize as retryable, user-fixable, or fatal, and do not suggest next steps (e.g., 'Check Modbus server is running', 'Verify unit ID'). This violates pattern:recovery-guide and pattern:error-classification.
Destructive operations (write_coil, write_coils, write_register, write_registers, mask_write_register) lack confirmation or dry-run capability. Agents making mistakes could inadvertently write to the wrong address or unit. No pattern:confirmation-request implemented.
Tool selection ambiguity: read_coils (address 1), read_discrete_inputs (address 10001), read_registers (address 40001) have overlapping semantics but inconsistent defaults. Tool descriptions do not clarify when to use each, Modbus register types (coils, discrete inputs, holding registers, input registers) require domain knowledge. LLMs may hallucinate tool calls without guidance.
Parameter 'data' in write_coil is typed as 'boolean|integer' without documentation of which is expected. JSON Schema does not cleanly express union types; Modbus coils expect 0x0000 or 0xFF00 (not arbitrary integers). This invites invalid input from LLMs.