Loading ToolBench
A Model Context Protocol server that provides access to HomeKit accessories and rooms, enabling management and control of HomeKit devices through MCP tools
12 tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. All tools have names starting with action verbs (get_, set_, rename_, accessory_) and descriptions present. However, parameter descriptions are minimal (1-2 words), output schemas are not documented, and error handling guidance is absent. Tool composition is reasonable (single responsibility), but parameter naming lacks type suffixes (e.g., 'name' vs 'accessory_name' inconsistency). No tool annotations (readOnlyHint/destructiveHint) despite clear read/write distinction in risk metadata. Schemas are present but sparse, most parameters lack detailed constraints, ranges, or format guidance.
Turn off lights/switches, close covers
Turn on lights/switches, open covers
Toggle accessory state
Find accessory by name
Get all HomeKit accessories with their names, rooms, and UUIDs
Get all HomeKit rooms with their names and UUIDs
Get room accessories
Output schemas not documented. LLMs cannot infer what fields are returned (e.g., does get_all_accessories return UUID, name, room, type, status?). This forces agents to guess and risks failed downstream tool calls.
Parameter descriptions are minimal (1-2 words: 'UUID of the accessory to move', 'Name of the accessory'). Missing format guidance, constraints, and examples.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk metadata exists in test file but not exposed in tool definitions. Agents cannot distinguish safe reads from destructive writes without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
Find room by name
Rename accessory
Rename room
Move an accessory to a different room
Move accessory by name
No error handling guidance. Tools lack recovery hints (e.g., 'If accessory not found, try get_all_accessories() to list available names'). Agents have no path forward on failures.
Parameter naming inconsistency: some tools use 'name' (get_accessory_by_name), others use 'accessory_name' (accessory_on). This ambiguity forces LLM reasoning overhead.