Loading ToolBench
A Model Context Protocol server for controlling Home Assistant smart home devices through tools for lights, climate control, locks, alarm panels, and humidifiers
This Home Assistant MCP server has 12 tools across 5 domains (light, climate, alarm_control_panel, lock, humidifier). While tool names follow verb_noun convention and schemas are present, there are significant quality gaps: (1) Descriptions are minimal (10-30 chars), well below the 50-200 char LLM-optimized baseline. (2) Parameter descriptions exist but are generic (e.g., 'The ID of the light to control' repeated across tools). (3) No output schema documentation, callers cannot predict response structure. (4) Copy-paste errors in parameter descriptions (alarm tools describe 'The ID of the light to control' instead of 'alarm control panel'). (5) Inconsistent schema format: 'int' vs 'integer' type in humidifier-set_temperature. (6) No error handling or recovery guidance. (7) No security/permission documentation. (8) No tool annotations (destructiveHint, readOnlyHint). Tool registration is visible in server.py via service.tools dictionaries, but tool definitions themselves are spread across service modules not shown in full. Based on visible evidence, the server is functional but falls well short of production quality.
Arm the alarm control panel entity
Disarm an alarm control panel entity
Set the temperature of a climate entity
Turn off a climate entity
Set the temperature of a humidifier entity
Turn off a humidifier entity
Turn on a humidifier entity
Minimal tool descriptions (10-35 chars), far below the 50-200 char LLM-optimized baseline. LLMs cannot determine when to call a tool vs. alternatives with such sparse context. Examples: 'Turn on a light entity' (23 chars), 'Arm the alarm control panel entity' (34 chars).
Copy-paste error: alarm_control_panel-disarm and alarm_control_panel-arm both describe entity_id as 'The ID of the light to control' instead of alarm control panel. This will confuse LLMs about what resource they are actually controlling.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 48 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 42 | - | v1 |
Set the brightness of a light entity
Turn off a light entity
Turn on a light entity
Lock a lock entity
Unlock a lock entity
No output schema documentation. Tools return JSON responses (visible in server.py TextContent wrapping), but callers have no schema to understand return structure. This forces LLMs to guess field names for chaining or data extraction.
No error handling guidance. When call_service() fails in server.py (line catches Exception and re-raises), the error is a raw exception string. LLMs receive no recovery hint, retry classification, or actionable next steps.
Generic parameter descriptions repeated across tools ('The ID of the light to control', 'The code to arm the alarm control panel'). While technically present, they do not distinguish tools or explain when to use one over another. Baseline: parameter descriptions should be 50-150 chars with context.
Inconsistent schema type definitions: humidifier-set_temperature uses 'int' instead of 'integer' (JSON Schema standard). This will confuse schema validators and LLM parsers expecting canonical JSON Schema.
No tool annotations. Tools lack destructiveHint, readOnlyHint, or idempotentHint metadata. All tools are WRITE operations (high risk), but LLMs have no explicit signal, they may retry indiscriminately, causing duplicate lock/unlock or temperature-set calls.
Credentials are environment variables (HOMEASSISTANT_TOKEN, HOMEASSISTANT_BASE_URL) rather than parameters, which is correct. However, there is no server-side validation to ensure these are present before tool execution, if missing, tools will fail with a raw exception rather than a configuration error.
No parameter validation or constraint documentation. E.g., brightness_pct and humidity are described as 0-100 ranges but schema does not enforce min/max. LLMs can pass invalid values (e.g., -10 or 150) and will receive opaque errors.
No idempotency documentation or guarantee. Tools like lock-lock, lock-unlock, light-turn_on are state-setting and should be idempotent, but there is no explicit contract or retry safety documentation for LLMs.