Loading ToolBench
MCP server for ShellyAdmin, a self-hosted Shelly fleet management system. Provides read-only and state-changing tools for device discovery, firmware management, compliance checking, and bulk operations on Shelly smart home devices.
ShellyAdmin demonstrates solid tool design with 21 well-named, verb-prefixed tools (list_, get_, execute_, etc.). All tools have descriptions (avg ~180 chars, within 10-1024 baseline). Input schemas are present and typed for all tools. However, output schemas are not documented in the visible code, critical for LLM planning. Parameter descriptions are present but some lack detail on constraints (e.g., 'target' accepts MAC/IP/name but doesn't specify format). Error handling is mentioned in descriptions (preview/confirm pattern) but recovery guidance is minimal. Security is strong: confirm-before-execute gates destructive ops, audit logging is explicit, and no secrets in params. Composition is excellent, tools chain well (list_devices → get_device → execute_device_action). The preview/confirm pattern is a standout for irreversible operations.
Apply a configuration action to multiple devices at once (e.g., set_timezone, set_ntp_server). Risk varies by action. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Fetch a summary of device compliance status across the fleet (e.g., firmware versions, configuration drift, security posture).
Register devices found by the most recent scan into the persistent inventory. Pass macs=[] to register everything pending; otherwise pass a subset. Risk: medium — adds rows to the device store and starts ongoing refresh / firmware probes against them. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Execute a single action on a device (e.g., reboot, factory_reset, switch toggle). Risk varies by action (see preview). OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Output schemas not documented in visible code. LLMs cannot plan downstream tool calls or extract required fields (e.g., what fields does list_devices return? what structure does get_device expose?). This forces agents to guess at response structure.
'target' parameter (used in get_device, list_device_actions, export_device, refresh_device, execute_device_action) accepts MAC, IP, or device name but lacks format constraints. Description should specify: 'MAC (XX:XX:XX:XX:XX:XX), IPv4 (dotted-quad), or device name (string, case-insensitive)'.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
Export a device's full state (device row + raw config + raw status + capabilities) as a single JSON document.
Query every known device's available firmware versions (stable + beta). Spawns a firmware_check job. Risk: low — read-only. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Install firmware on a set of devices. Spawns a firmware_install job. Risk: high — devices will reboot and may be temporarily unavailable. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Poll the status of the most recent firmware install job. Returns per-device installation progress and results.
Poll the status of the most recent firmware check job. Returns per-device firmware availability with optional filtering and paging for large fleets.
Fetch full detail (config, status, capabilities, available actions) for a single device.
Query the audit log with optional filtering by level, message substring, or time range. Returns paginated results.
Fetch the current ShellyAdmin configuration (subnets, timeouts, firmware channels, etc.). The MCP token is redacted.
Fetch a single configuration template by name.
List all saved device credentials (usernames and tags only; passwords and HA1 hashes are redacted).
List the actions (reboot, factory_reset, ota_revert, switch toggles, etc.) supported by a specific device.
List Shelly devices known to ShellyAdmin. Optional search/gen/limit filters, and a `fields` allowlist to return only the columns you need — the unfiltered shape runs to 59 keys per device (~50 KB for 44 devices) and overflows MCP output caps.
List all saved device configuration templates.
Re-probe every known device. Spawns a background refresh job. Risk: low — read-only with respect to devices. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Re-probe a single device (HTTP+RPC) and update its stored snapshot. Risk: low — read-only with respect to the device, only ShellyAdmin's local cache changes. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Poll the status of the most recent network scan. Returns pending discoveries (slim per-device summary) and job metrics.
Kick off a network scan over the configured subnets. Discovered devices land in scan_status.pending; nothing is registered until confirm_scan is called. Risk: low — read-only with respect to devices on the network. OPERATOR APPROVAL REQUIRED. This tool changes device state. The first call MUST omit confirm (or set it to false) to receive a structured preview describing what would happen. Surface the preview to the operator in plain language and obtain explicit yes/no approval. Only after the operator says yes may you call again with confirm=true to execute. Each preview/execute call is audit-logged with the request_id so the operator can verify what ran.
Error handling descriptions mention preview/confirm pattern but lack recovery guidance. E.g., if firmware_install fails, what should the agent do next? Retry? Call firmware_status? Call get_logs? Descriptions should include 'If this fails, try: ...'.
Pagination parameters (limit, offset) present in firmware_status and get_logs but absent from list_devices despite description mentioning '~50 KB for 44 devices' overflow risk. list_devices should enforce a default limit (e.g., 20) and require explicit pagination for larger results.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) not visible in schema. All 13 read-only tools should declare readOnlyHint=true; destructive tools (firmware_install, execute_device_action with factory_reset) should declare destructiveHint=true; idempotent tools (refresh_device, get_device) should declare idempotentHint=true. This enables smarter agent planning.