Loading ToolBench
Safely manages only the isolated Home Assistant AI automation file.
Strong foundation with 15 well-named tools using clear verb_noun patterns (create_, update_, delete_, list_, get_, enable_, disable_, trigger_, validate_, reload_, diff_, rollback_). All tools have descriptions (avg 95 chars, within 10-1024 baseline). Input schemas present for all tools with type annotations. However, parameter descriptions are minimal or absent in many cases, most params lack guidance on format, constraints, or dependencies. Output schemas are not documented. Error handling is implicit (no recovery guidance visible). Security posture is good (no secrets in params, audit logging configured), but tool composition could be tighter (some tools like update_ai_automation accept both replacement and patch, creating ambiguity).
Create an isolated automation transactionally; defaults to disabled.
Delete one AI-managed automation transactionally.
Diff the current AI file against a candidate or a named backup.
Disable a loaded automation whose id exists in ai_automations.yaml.
Enable a loaded automation whose id exists in ai_automations.yaml.
Return one complete AI-managed automation by its stable id.
Parameter descriptions missing or minimal. Most params (automation_id, automation, reason, backup_filename, honor_conditions) lack format guidance, constraints, or dependency hints. E.g., 'automation_id' has no guidance on format (UUID? slug? alphanumeric?), and 'automation' accepts 'dict|string' but no description of expected YAML/JSON structure.
Output schemas not documented. Tool descriptions state what they return (e.g., 'Return one complete AI-managed automation') but do not specify the structure, field names, or types. LLMs cannot plan downstream calls without knowing response shape.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
Return MCP, Home Assistant API, validation, and AI file health.
List safe timestamped backups and their hashes.
List only automations stored in ai_automations.yaml.
Validate and normalize an automation without changing any file.
Reload Home Assistant automations without restarting Home Assistant.
Restore a named backup transactionally, after backing up current state.
Trigger an AI-managed automation through Home Assistant.
Replace or shallow-patch one AI-managed automation transactionally.
Run Home Assistant's documented full configuration validation API.
Ambiguous parameter relationships in update_ai_automation. Tool accepts both 'replacement' (complete replacement) and 'patch' (shallow patch) but does not document mutual exclusivity or precedence. LLMs may pass both, causing undefined behavior.
No error recovery guidance. Tool descriptions do not indicate what errors are possible, whether they are retryable, or what the LLM should do next. E.g., 'automation not found' vs 'validation failed' vs 'Home Assistant unreachable' all require different recovery strategies.
Tool annotations missing. No readOnlyHint, destructiveHint, or idempotentHint attributes visible in tool definitions. This prevents LLMs from understanding which tools are safe to retry, which are destructive, and which are read-only without parsing descriptions.