Loading ToolBench
MCP server for direct HTTP communication with air-Q air quality sensor devices
air-Q has well-structured tool definitions with clear naming conventions and generally good descriptions. All 15 tools start with action verbs (get_, list_, set_, plot_, export_, restart_, shutdown_) and parameter descriptions are present. However, there are significant gaps: output schemas are not documented anywhere in the codebase, error handling guidance is minimal, and several design patterns from the 54 Agentic Tool Patterns are not fully implemented. Tool responses are not shaped or constrained, the server returns raw API responses without documented field structure. Parameter validation and recovery guidance are absent. The risk annotations (READ_ONLY, WRITE, DESTRUCTIVE, IRREVERSIBLE) are present in metadata but not exposed via toolAnnotations in the MCP response where LLMs can see them.
Configure network settings. Set dhcp=True for DHCP, or provide ip/subnet/gateway/dns for static IP. After changing network settings, the device must be restarted.
Export air quality history as CSV or JSON. Data is column-oriented with compound sensor values split into separate columns.
Get current air quality measurements from a device, including indices, climate, gases, particulates, and metadata.
Retrieve air quality history from device storage. By default returns the last 1 hour. Optionally filter by sensor(s). History is returned as row-oriented JSON.
Get device configuration. Use get_config_guide prompt for interpretation of config keys.
Get device information including serial number, firmware version, model, and other metadata.
Output schemas are not documented. No tool in the codebase declares what fields or structure LLMs should expect in responses. Baseline rubric requires 100% of A+ tools to have documented return types. This forces LLMs to parse unstructured responses and breaks downstream tool chaining.
Enum constraints are written as examples in descriptions (e.g. 'Common themes: standard, CO2, VOC, Humidity, PM2.5, Noise') rather than declared in JSON Schema. LLMs treat examples as suggestions, not exhaustive options, risking invalid values like 'rainbow' or 'custom_theme'. Rubric requires enums for known value sets.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 48 | - | v1 |
Get list of available LED theme names that can be set on the device.
List all configured air-Q devices with their addresses and configured metadata.
Plot air quality history data. Returns PNG or HTML visualization.
Restart a device. It will be unreachable for about 30 seconds.
Set LED brightness. 'default' is the normal brightness (0-100%), 'night' is optional night brightness.
Rename a device. The new name appears on the device display.
Set the LED visualization theme for one or both sides of the device. Common themes: 'standard', 'CO2', 'VOC', 'Humidity', 'PM2.5', 'Noise'. Use get_possible_led_themes to see all available themes.
Configure night mode. Times in 'HH:mm' format (UTC). brightness_day/brightness_night are percentages (0-100). fan_night_off disables the particle sensor fan at night. wifi_night_off caches data to SD and uploads when wifi returns. alarm_night_off disables acoustic warnings (fire/gas still trigger).
Shut down a device. It must be manually powered on again. Only use if explicitly requested.
Risk annotations (READ_ONLY, WRITE, DESTRUCTIVE, IRREVERSIBLE) are stored in Python metadata but not exposed via MCP toolAnnotations. The MCP spec supports readOnlyHint and destructiveHint to signal to LLMs which tools are safe to retry and which have irreversible effects. Agents cannot reason about retry safety without this info.
No error handling guidance. Tools that fail (e.g. device not found, network timeout, auth failure) return errors but provide no recovery hints. Rubric requires errors to tell the LLM what to do next: 'Device not found. Try list_devices() to see available devices.' Current error responses (inferred from Python error handling) are silent.
Mutual exclusivity of parameters is not formalized. configure_network requires dhcp=true for DHCP or (ip, subnet, gateway, dns) for static IP, but the schema does not declare this constraint. LLMs may pass both or neither. Rubric requires explicit dependency documentation in parameter descriptions AND validation inside the tool.
Date/time parameter formats are described in text (e.g. 'ISO 8601 datetime string') but not constrained by regex or format declaration in JSON Schema. LLMs frequently misformat dates. Rubric requires formal constraints (format: 'date-time') in schemas, not just text descriptions.
Device parameter allows optional substring matching (e.g. 'Living' matches 'Living Room'), but this behavior is not formally defined. If two devices match, the error message and fallback are unclear. Rubric requires clear documentation of lookup behavior: exact match only, or prefix match? What if ambiguous?
set_night_mode time parameters require 'HH:mm' format (e.g. '22:00') but constraint is in description, not schema. No regex pattern or minLength/maxLength enforcement. LLMs may pass '22', '22h00', or '10:00 PM'. Rubric requires formal constraints.
Numeric parameters (brightness_day, brightness_night, default, night in set_brightness) state range 0-100 in descriptions but likely lack minValue/maxValue in JSON Schema. Unbounded numeric inputs let LLMs pass -50 or 500, causing API errors.
get_config description says 'Use get_config_guide prompt for interpretation of config keys' but this prompt is not visible in the provided code samples. If the prompt does not exist, this creates a broken reference that confuses LLMs.