Loading ToolBench
A read-only MCP server that wraps the Shelly Cloud Control API as curated, well-described tools for the Hermes agent. Exposes electricity consumption data from Shelly devices.
Shelly MCP demonstrates solid fundamentals: all 7 tools have clear verb-noun naming (list_devices, get_power, etc.) and substantive descriptions (avg 180 chars, well above 10-char minimum). Input schemas are present and typed for all tools. However, critical gaps prevent a higher score: (1) parameter descriptions are minimal or absent, most params lack actionable guidance on format/constraints; (2) output schemas are undocumented, callers cannot see what fields to expect; (3) error handling is generic (RuntimeError with message) rather than classified/actionable; (4) no pagination support despite tools returning potentially large result sets (e.g., get_consumption_history, get_net_energy_history). Tool composition is sound (each does one thing), and naming is consistent and clear. The server is production-ready for read-only consumption queries but lacks the polish expected of A-grade tooling.
Get historical power consumption data in time-bucketed intervals. Supports named date ranges (day, week, month, year) or custom date ranges. Returns consumption statistics for each bucket.
Get a summary of current power consumption and cumulative energy for one or all devices. Combines live power (watts) and total energy (kWh) in a single call.
Get the raw device status payload from the Shelly Cloud API for a single device. Returns the complete status object including all components, their state, and any reported errors.
Get cumulative energy consumption (in kWh) for one or all devices. Aggregates energy counters from device components (em:N, switch:N, pm1:N, emeters[]). Missing/null readings are reported in warnings rather than silently counted as zero.
Get historical net energy (import/export) data in time-bucketed intervals. Supports named date ranges (day, week, month, year) or custom date ranges. Returns net energy statistics for each bucket.
Output schemas are completely undocumented. Callers cannot see what fields to expect from any tool response. This forces LLMs to guess at response structure and breaks downstream tool chaining.
Parameter descriptions are minimal or absent. The 'device_id' param in get_power, get_energy, get_consumption_summary says 'Optional device ID. If omitted, returns power for all devices.' but lacks format guidance (is it a string UUID? a numeric ID?). Date range params in get_consumption_history and get_net_energy_history lack examples of valid datetime format beyond the description text.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Get current active power consumption (in watts) for one or all devices. Returns live power readings across device families (gen2 energy meters, switches, plugs, gen1 meters). Missing/null readings are reported in warnings rather than silently counted as zero.
List all Shelly devices in the account with their metadata (id, name, model, generation, online status).
No pagination support. Tools returning lists (list_devices, get_consumption_history, get_net_energy_history) lack limit, offset, or cursor parameters. Large result sets will blow context windows and degrade LLM reasoning.
Error handling is generic. All errors are RuntimeError with a message string. No error classification (retryable vs. user-fixable vs. fatal), no structured error responses, no recovery guidance. LLMs cannot distinguish 'auth failed' from 'rate limited' from 'device offline'.
Date range parameter constraints are under-specified. get_consumption_history and get_net_energy_history accept 'custom' date_range but the format string 'YYYY-MM-DD HH:MM:SS' is only in the description, not enforced as a regex pattern in the schema. LLMs may pass invalid formats.