Loading ToolBench
Server has 10 tools with basic naming and descriptions, but significant gaps in schema completeness and parameter documentation. All tools have descriptions (10-100+ chars), but most lack detailed parameter descriptions and output schema documentation. No tool annotations (readOnlyHint/destructiveHint). Parameter schemas are minimal, most tools accept only string parameters with sparse descriptions. Error handling and recovery guidance are absent. Naming follows verb_noun convention (list_, get_, set_, run_, control_) which is good, but descriptions lack WHEN/WHY context and downstream chaining information.
High-level device control with common commands. Supported commands: 'on', 'off', 'toggle', or property assignments like 'brightness=50', 'color_temperature=4000'. For lights: 'on', 'off', 'toggle', 'brightness=<0-100>', 'color_temperature=<value>' For switches/plugs: 'on', 'off', 'toggle'
Get all available properties and their current values for a device by its name.
Get a specific property value from a device. Use list_device_capabilities to see available properties.
List all available properties and actions for a device.
List all devices in the Mi Home account. Optionally filter by home_id.
List all homes in the Mi Home account.
Missing output schema documentation. No tool documents what fields are returned, their types, or structure. LLMs cannot plan downstream calls or extract required data (e.g., device_id for chaining).
Parameter descriptions are minimal or missing. Most parameters have only a brief label (e.g., 'Device name', 'Property name') with no format, constraints, or examples. LLMs cannot validate input or understand edge cases.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk field is present in metadata but not exposed in tool definition. LLMs cannot distinguish safe reads from destructive writes without explicit hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | <=2025-11-25 | v2 |
List all available scenes/automations in Mi Home.
Run an action on a device (e.g., 'toggle', 'start-sweep'). Use list_device_capabilities to see available actions.
Run a scene/automation by its ID. Use list_scenes to get available scene IDs.
Set a property value on a device. The value will be automatically converted to the appropriate type (bool, int, float, or string).
No error handling or recovery guidance. Tools lack descriptions of failure modes, invalid inputs, or what the LLM should do if a call fails (retry, ask user, try alternative tool).
control_device accepts free-form command strings without enum constraints. Description lists examples ('on', 'off', 'brightness=50') but LLMs may hallucinate invalid commands. Should use enum or strict validation.