Loading ToolBench
Strong schema definitions with proper JsonSchema derivation and enum constraints. All 21 tools have descriptions and typed parameters. However, descriptions are terse (avg ~60 chars, below the 194-char baseline), lacking WHEN/WHY context and recovery guidance. No output schemas documented. Composition issues: tools like 'create_and_label_device' and 'create_and_assign_label' violate single-responsibility (pattern:tool). Error handling is minimal, no recovery guidance or actionable error messages visible in code. Security: credentials injected via config (good), but no audit logging or permission gates for destructive ops (delete, update).
Create a new label and assign it to multiple entities
Create a new label and apply it to a device
Create a new label in Home Assistant
Get detailed information about a Home Assistant resource type
Find devices in Home Assistant with optional filtering by area or manufacturer
Get detailed information about a specific device
Get all entities associated with a device
Composite tools violate single-responsibility principle. 'create_and_label_device' and 'create_and_assign_label' combine two distinct operations (create + assign), forcing agents to invoke them as atomic units when they may want to create without labeling or label without creating.
Descriptions are terse (avg ~55 chars, well below 194-char baseline). Most lack WHEN/WHY context, prerequisites, or recovery guidance. E.g., 'Update entity metadata' does not explain when to use entity_update vs entity_assign_area vs entity_add_labels.
No output schemas documented. LLMs cannot infer what fields are returned by each tool, forcing them to guess downstream parameter names and risking broken tool chains.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | <=2025-11-25 | v2 |
Apply labels to all entities of a device
Update device metadata (name, area)
Add labels to an entity (preserves existing labels)
Assign an entity to an area
Find entities in Home Assistant with optional filtering by domain, area, device, or labels
Get detailed information about a specific entity
Remove labels from an entity
Update entity metadata (name, area, disabled status, labels)
Get the current state of all entities in Home Assistant
Create a new item in a Home Assistant registry
Delete an item from a Home Assistant registry
List items from a Home Assistant registry (areas, devices, entities, floors, labels, or state objects)
Update an item in a Home Assistant registry
Update an existing label in Home Assistant
No error handling guidance visible. Destructive operations (registry_delete, entity_update) lack confirmation patterns or recovery instructions. No actionable error messages (e.g., 'Invalid resource type: must be one of area_registry, device_registry, ...').
No permission gates or audit logging for destructive operations. Agents can delete registries or update entities without explicit authorization checks or traceable logs.