Loading ToolBench
io-core — a safe I/O layer for LLM agents (MCP server: serial, Modbus TCP, MQTT, file sandbox, ESP32 flash tools)
ironharness demonstrates solid tool definition quality with consistent naming patterns (verb_noun: serial_open, mqtt_publish, file_read), comprehensive parameter descriptions, and explicit JSON Schema input definitions across all 24 tools. Tool descriptions are substantive (avg ~150 chars) and explain purpose, prerequisites, and return behavior. However, output schemas are not formally documented in the visible code, responses are described in text but lack structured schema declarations. Error handling is domain-aware (custom exception mapping via _domain_errors decorator) but lacks recovery guidance in tool descriptions. Security posture is strong (sandbox isolation, environment-based configuration, no credential parameters), but descriptions could better explain risk levels and when tools require special permissions (e.g., esp_flash requires IRONHARNESS_ALLOW_REAL_FLASH=1).
Returns the text back. Service tool to verify the MCP setup works.
Erases the flash on a live ESP32 board (requires IRONHARNESS_ALLOW_REAL_FLASH=1).
Flashes an ESP32 firmware image to a live board (requires IRONHARNESS_ALLOW_REAL_FLASH=1).
Parses an ESP32 firmware image and returns metadata (chip, entrypoint, segments, flash mode/size/freq).
Deletes a file from the sandbox.
Lists files in a sandbox directory.
Reads a file from the sandbox; returns hex-encoded bytes.
Output schemas not formally documented. Tool descriptions explain return behavior in prose (e.g., 'returns hex string', 'returns metadata'), but no structured JSON Schema output definitions are visible in the code. LLMs cannot reliably parse unstructured responses or plan downstream tool chains without explicit output schemas.
Destructive operations (file_delete, esp_flash, esp_erase) lack confirmation/dry-run guidance in descriptions. Descriptions state what the tool does but do not warn agents about irreversibility or suggest a confirmation pattern. Agents may invoke these without understanding consequences.
Permission requirements not documented in tool descriptions. esp_flash and esp_erase require IRONHARNESS_ALLOW_REAL_FLASH=1 environment variable, but this constraint is not mentioned in the tool descriptions. Agents will not know why calls fail or how to enable them.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Writes bytes to a file in the sandbox; data_hex is a hex string.
Closes a Modbus TCP connection.
Opens a Modbus TCP connection to a host:port.
Reads holding registers from a Modbus device (FC3 query).
Writes holding registers to a Modbus device (FC16 query).
Closes an MQTT connection.
Opens an MQTT connection to a broker.
Publishes a message to an MQTT topic.
Reads a message from an MQTT subscription.
Subscribes to an MQTT topic.
Closes a named serial port.
Lists host serial ports with USB identity: device, vid, pid, serial_number, location, description. Board COM numbers float across re-plugs — enumerate before serial_open. Nothing is opened or consumed.
Opens a named serial port (COM3, /dev/ttyUSB0, loop:// for tests).
Reads up to size bytes from the serial port; returns a hex string ("" — timeout).
Blocks until a serial port matching the given VID:PID appears, then returns {"device": ..., "serial_number": ...}. Use when the board's COM number floats across re-plugs. Raises TimeoutError if nothing matches.
Writes bytes to the serial port; data_hex is a hex string (e.g. 48656c6c6f).
Session introspection: open transports (name, kind, port), background readers (buffered bytes, alive), in-flight transfers, running serial monitors, sandbox root. Call after context loss to self-recover instead of guessing.
Error recovery guidance missing. Domain exceptions are mapped to ToolError with text (via _domain_errors decorator), but tool descriptions do not include recovery hints. E.g., serial_wait timeout should suggest 'Check VID:PID or increase timeout parameter'; PolicyViolation should suggest 'Check IRONHARNESS_ALLOWED_HOSTS configuration'.
Hex encoding/decoding not explained in parameter descriptions. serial_write, file_write, mqtt_publish accept 'data_hex' parameters, but descriptions do not clarify the hex format (e.g., '48656c6c6f' for 'Hello'). Agents may pass raw bytes or ASCII instead of hex-encoded strings.