Loading ToolBench
FastAPI server for home automation system with MCP tools to control smart home devices (lights, thermostats, locks, blinds, fans, garage doors, sprinklers, EV chargers) and set home modes.
The server defines 3 tools with partial schemas and descriptions. All tools have names starting with action verbs and descriptions present, but critical gaps exist: parameter descriptions are minimal or missing for key inputs, output schemas are not documented, error handling guidance is absent, and no input validation is apparent. The tool definitions appear to be inferred from code rather than formally registered with complete metadata. STDIO transport adds a hard ceiling of 50, but definition quality itself scores marginally better only due to basic naming and description presence.
Universal device control tool. Control smart home devices by ID, room, or type.
Query device states and information.
Open the garage door.
Parameter descriptions missing or minimal across all tools. 'room', 'device_type', 'brightness', 'position', 'speed', 'target_temp', 'mode' lack actionable guidance for LLM.
No output schemas documented. LLMs cannot infer what fields get_device_status or control_device return, breaking tool chaining and context planning.
No error handling guidance. If a device is not found, the tool likely returns a generic error; descriptions do not guide LLM recovery (e.g., 'Try get_device_status first to discover available devices').
Mutual exclusivity of parameters (device_id vs room vs device_type) undocumented in control_device. LLM may pass multiple, causing ambiguous behavior.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 54 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
open_garage has no confirmation or dry-run step. Agents may call it unintentionally, opening a garage without user consent, a destructive action with no undo.
No pagination declared for get_device_status. Returning all devices could blow context window; missing limit parameter and result count.
control_device description does not declare state-change. LLM cannot distinguish whether calls are idempotent or have side effects, complicating retry logic.
No permission or scope hints. Tools that control physical devices (open_garage, control_device) do not declare required permissions (e.g., 'requires write:devices').