Loading ToolBench
Server has 8 tools with clear naming (verb_noun pattern: ipmi_read, ipmi_exec, ipmi_power_status, etc.). Descriptions are present and substantive (100-300 chars), explaining WHAT each tool does and WHEN to use it. However, most tools lack explicit input parameter schemas in the visible code, parameters are inferred from function signatures rather than declared in JSON Schema. Tool annotations are present (readOnlyHint, destructiveHint) which is excellent. Error handling is strong: ipmi_exec implements a multi-level confirmation pattern (confirm + force flags) with clear refusal messages. Security is well-designed: destructive operations require explicit confirmation, host-critical actions require force=true on top. Audit logging is implemented. Main gaps: (1) no visible JSON Schema definitions for input parameters, (2) output schemas are not documented, (3) parameter descriptions are embedded in docstrings rather than in schema metadata.
Detailed chassis status (power, fault, intrusion…).
Run an arbitrary ipmitool subcommand against the BMC. Security (fail-safe): - a recognized read subcommand runs directly; - any mutating subcommand is NOT executed without ``confirm=true``; - a host-critical action (power off/cycle/reset/soft, mc reset) requires ``force=true`` ON TOP. Without confirmation, the tool only returns a preview of what would run.
Management controller info (firmware version, IPMI version…).
Current chassis power state (ipmitool chassis power status).
Run a READ-ONLY ipmitool subcommand against the BMC (e.g. "sensor list"). Refuses any subcommand not recognized as safe. For a mutating action, use ``ipmi_exec`` (which will require confirm=true).
(Re)build the local SDR cache used to speed up sensor reads. Runs ``sdr dump <IPMI_SDR_CACHE>``. This one call is slow (it reads the whole SDR repository from the BMC); afterwards, sensor reads reuse the cache via ``-S`` and are fast. Requires IPMI_SDR_CACHE to point at a writable path (mount a volume to persist it across container runs).
No visible JSON Schema definitions for tool input parameters. Parameters are inferred from Python function signatures (e.g., ipmi_read(command: str)) rather than declared in structured JSON Schema with type, description, and constraints. This prevents clients from validating inputs before sending and makes parameter intent opaque to LLMs.
Output schemas are not documented. Tools return plain strings (e.g., 'ipmitool chassis power status\n(exit=0, 0.123s)\n--- stdout ---\nChassis Power is on'). LLMs cannot parse structured output or extract fields for downstream tool calls. No documented return type or field structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
List the System Event Log (ipmitool sel elist).
List all sensor readings (temperatures, fans, voltages). Uses ``sdr elist`` — far cheaper than ``sensor list`` on a high-latency link. Call ``ipmi_refresh_sdr_cache`` once (with IPMI_SDR_CACHE set) to make this near-instant even over a VPN.
ipmi_exec's 'command' parameter accepts arbitrary ipmitool subcommands with no enum constraint or format validation in the schema. The tool implements runtime classification (classify(command)) to reject unsafe commands, but this is not declared in the schema. LLMs cannot discover valid command patterns upfront.
No pagination or result limits documented. ipmi_sensors and ipmi_sel return raw ipmitool output which could be very large on systems with many sensors or events. No documented limit, no pagination guidance, no truncation strategy.
Parameter descriptions are embedded in docstrings, not in schema metadata. The 'confirm' and 'force' parameters in ipmi_exec have descriptions in the Python docstring but are not visible in the JSON Schema that clients receive. This breaks schema-driven UI generation and parameter discovery.