Loading ToolBench
Single tool 'nixos_ops' with comprehensive description but critical schema and parameter documentation gaps. Tool name lacks action verb (should be 'manage_nixos' or similar). Input schema visible in code uses Zod with enums and descriptions, but parameter descriptions are minimal and lack actionable constraints. Output is unstructured text, not typed fields. No documented return schema. Error handling returns generic text without recovery guidance. Safety model is well-described but not enforced via parameter validation or confirmation patterns.
Remote NixOS system management via SSH. Reads are accelerated by NATS cache when available. Actions: list-nodes — Show configured nodes, tags, and permission flags status — System health: version, uptime, generation, K3s, ZFS, memory, disk, failed services rebuild-dry — Safe dry-build (no changes applied) rebuild-switch — DESTRUCTIVE: activate new config. Requires allowRebuildSwitch=true on node, confirm=true, AND ZFS preflight pass generations — List NixOS generations rollback — DESTRUCTIVE: revert to previous generation. Requires allowRollback=true AND confirm=true validate — Syntax check and evaluation of NixOS config zfs-preflight — Check ZFS pool health, import status, usage, scrub status diff — Build pending config and compare to current (uses nvd/nix-diff if available) Safety model: - rebuild-switch has THREE gates: node permission, confirm flag, and ZFS preflight - rollback has TWO gates: node permission and confirm flag - All commands have timeouts (120s default, 600s for builds) - Never uses --flake unless the node has useFlake=true
Tool name 'nixos_ops' lacks action verb. Should be 'manage_nixos_system' or split into 'get_nixos_status', 'rebuild_nixos_config', 'rollback_nixos_generation' to match verb_noun pattern and enable LLM to infer intent from name alone.
Output is unstructured plain text (formatResult returns raw string). LLMs cannot parse structured fields for downstream tool chaining. Should return typed JSON with fields like {status: string, version: string, uptime_seconds: number, failed_services: string[]} for status action, enabling agents to extract and reuse specific values.
Parameter 'node' description says 'required except for list-nodes' but is marked optional in schema. Zod schema shows optional() but no conditional validation. LLM cannot determine when node is truly required. Add explicit validation: if action !== 'list-nodes' && !node, return error 'node parameter required for action X'.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
Destructive operations (rebuild-switch, rollback) lack confirmation pattern enforcement. Code checks confirm flag but does not enforce it at schema level or return a confirmation request (MRTR pattern). Should return {type: 'input_required', input_schema: {confirm: boolean}} if confirm=false on destructive action, letting LLM ask user before proceeding.
Error messages are generic text ('ERROR: ...') without recovery guidance. Per pattern:recovery-guide, errors should suggest next steps. E.g., 'Node not found: try action=list-nodes to see available nodes' or 'ZFS preflight failed: check pool health with action=zfs-preflight before rebuild-switch'.