Loading ToolBench
MCP server for controlling a Shelly Duo RGBW light with color presets and status indicators
The server has 4 clearly defined tools with basic name clarity, moderate descriptions, and mostly complete input schemas. Naming follows verb-noun convention (set_*, turn_*, get_*). Descriptions exist but lack WHEN/WHY context needed for LLM selection. Parameter descriptions are present but terse. Output schemas are undocumented, the LLM must infer response structure from tool result text. Error handling returns generic error strings without recovery guidance. The server implements simple color/status control for a hardware light; the domain is well-scoped but the tool definitions lack production-grade polish. No tool annotation hints (readOnlyHint/destructiveHint) despite clear read vs. write semantics. Average per-tool score is ~62, placing this in the C+ range, functional but with noticeable gaps in LLM-optimized descriptions and structured output documentation.
Get the current status of the Shelly Duo RGBW light (on/off, color, brightness).
Set the Shelly Duo RGBW light to a specific color and turn it on. Supports named colors (red, green, blue, yellow, purple, magenta, cyan, orange, pink, warm_white, cool_white, white) or hex codes (#FF0000).
Set the light to a predefined availability status. Use this when the user says things like 'I'm on a call', 'I'm available', etc.
Turn off the Shelly Duo RGBW light.
Output schemas undocumented. Tools return text strings (e.g., 'Light set to red at 100% brightness') but LLMs cannot parse structured data from these responses. Downstream tools or agent reasoning will struggle to extract actionable fields like current_color, current_brightness, or rgb_values.
Error messages lack recovery guidance. When 'Failed to set color' is returned, the LLM has no hint about why it failed (invalid color name? network timeout? device offline?) or what to do next. Should provide actionable error classifications like 'color_not_supported, valid colors are: red, green, blue, ..., try one of these' or 'device_unreachable, check if the Shelly device is online and accessible'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 62 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 46 | - | v1 |
No tool annotations. set_light_color and turn_off_light are clearly destructive (modify device state); set_status is destructive; get_light_status is read-only. Missing destructiveHint and readOnlyHint annotations that would help clients make correct decisions about when to call tools and whether to require confirmation.
turn_off_light has no parameters and a 50-character description. Description is too brief; it should explain when to call this vs set_status('off'). Why two separate tools? 'Turn off the Shelly Duo RGBW light.' is clear but generic, lacks context about when this is the preferred operation.
set_light_color and set_status descriptions include example values inline ('red, green, blue, ...') rather than as enums. This violates the constraint principle, LLMs may treat examples as the only valid options, or invent variations like 'light red' or 'dark red' that don't exist. The 'status' param in set_status has valid values documented in the description (on_air, available, busy, normal, off) but not as a JSON Schema enum constraint.
get_light_status output is a single unstructured text string ('Light is on | Color: R=255 G=0 B=0 W=0 | Brightness: 100%'). If a downstream tool or agent action needs to filter by brightness or compare colors, it must parse text. Structured JSON output (is_on: bool, color_rgb: {r, g, b, w}, brightness: number) would be idiomatic for agents.
No idempotency guarantees documented. If set_light_color is called twice with the same color and brightness, does it succeed both times (idempotent) or fail the second time (not idempotent)? If network failure causes a retry, agents need to know if a duplicate call will cause harm. Declare idempotency explicitly in descriptions or via idempotentHint annotation.