Loading ToolBench
An MCP server that provides tools to query and control Home Assistant smart home devices, including lights, media players, climate control, and other IoT devices via the Home Assistant API
This server has significant quality gaps across naming, descriptions, schemas, and error handling. While tool names follow verb_noun convention (hass_get_state, hass_set_state, hass_play_music), descriptions are inconsistent, some are lengthy and include untranslated Chinese, parameter descriptions are minimal or missing, input schemas lack complete type coverage, and error handling is generic without recovery guidance. The server is functional but falls well below production-grade baselines.
查询 Home Assistant 中设备的状态，包括灯光亮度、颜色、色温，媒体播放器的音量等。 Args: entity_id: Home Assistant 中的设备 entity_id (例如 "light.living_room") Returns: 包含设备状态和属性的字典，或错误信息
在指定媒体播放器上播放音乐或有声书 Args: entity_id: 媒体播放器的entity_id (如 "media_player.living_room_speaker") media_content_id: 音乐/有声书 (专辑名/歌曲名/艺术家中英文都可以或 "random") Returns: 包含操作结果的字典 已知设备: 书房,音响,media_player.shu_fang_2
设置智能家居设备状态 Args: entity_id: 设备ID (如: light.bedroom, media_player.living_room) target: 操作目标参数，使用 HassTargetModel 验证 Returns: Dict[str, Any]: 操作结果，包含状态和可能的错误信息 Example: >>> # 设置灯光亮度为50% >>> hass_set_state( ... entity_id="light.bedroom", ... target={"type": "brightness_value", "input": 50} ... ) >>> >>> # 设置RGB颜色 >>> hass_set_state( ... entity_id="light.kitchen", ... target={"type": "set_color", "rgb_color": [255, 100, 0]} ... ) >>> # 设置空调温度 >>> hass_set_state( ... entity_id="climate.ac", ... target={"type": "set_temperature", "input": 26} ... )
Inconsistent and vague tool descriptions. hass_play_music includes Chinese text, device-specific examples ('书房,音响'), and lacks clarity on when to use vs other tools. Descriptions do not consistently state WHAT the tool does, WHEN to use it, and what it returns. Example: 'Returns: 包含操作结果的字典' (returns dict with result) is too generic.
Missing or incomplete parameter descriptions in input schemas. The 'media_content_id' parameter in hass_play_music has a description but includes mixed-language examples. In hass_set_state, the 'target' parameter description is lengthy and includes inline examples ('e.g. light.bedroom') which LLMs may reuse literally. Parameter descriptions should avoid example values and use enums/patterns instead.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 7 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 48 | - | v1 |
Input schema for hass_set_state 'target' parameter uses a complex object with nested conditional fields (input, is_muted, rgb_color only apply to specific types). These dependencies are documented in the enum description but not explicitly in the schema. LLMs may pass irrelevant fields for the selected operation type. The schema should enforce field inclusion based on the 'type' value.
Error handling lacks recovery guidance. All three tools return generic error dictionaries (e.g., {'success': False, 'error': 'control failed: ...'}) without telling the LLM what to do next, whether to retry, or which alternative tools to try. Example from hass_play_music: 'playback failed, entity_id:..., status code: 404' does not guide the agent toward recovery (try discovering devices first? Invalid entity_id?).
Output schemas are not documented. No tools include a documented return type schema. hass_get_state returns {'success': bool, 'state': str, 'attributes': str} but this is inferred from the code, not documented in the tool definition. Agents cannot plan downstream calls or extract structured data without knowing the expected response shape.
Secrets are exposed in source code. The conf.py file contains a hardcoded API key ('yourtoken') and base_url in plain text. This is a critical security violation, credentials should use environment variables or vault injection, never appear as config literals or tool parameters.
No permission gating or audit logging visible in tool implementations. Tools execute Home Assistant API calls without checking if the calling agent/user has authority. No attempt to log who called what with which parameters. For destructive operations like hass_set_state, this is a compliance risk.
Input validation is incomplete. hass_set_state accepts an 'input' parameter (0-100) for brightness but 'set_temperature' also uses 'input' with a different range (17-30). The Pydantic model defines 'input' twice (line: 'input: Optional[int] = Field(None, ge=0, le=100)' and 'input: Optional[int] = Field(None, ge=17, le=30)'), only the last definition applies, silently breaking brightness validation. This is a silent bug, not a validation framework issue.
Naming clarity: hass_set_state is ambiguous. The tool accepts many operation types (turn_on, turn_off, brightness_value, volume_set, set_temperature, set_color, pause, continue, volume_mute) but the name does not hint at this breadth. An LLM seeing 'hass_set_state' might assume it sets a generic state, not that it controls lights, climate, media players, etc. Better names might be hass_control_device or separate tools like hass_set_light, hass_set_climate, hass_set_media.