Triage failing DeviceCloud Maestro test runs from your AI assistant — fail reasons, failure screenshots, and flaky-flow analytics, without opening the web console
Two tools with strong, action-oriented names (diagnose_run, suite_health) and comprehensive descriptions (180+ chars each). Input schemas are well-defined with proper types, enums, and descriptions. However, output schemas are not documented in the visible code, only input validation is shown. Error handling is present but generic (HTTP status + body). No tool annotations (readOnlyHint/destructiveHint). Parameter descriptions are clear and include defaults, but no explicit guidance on chaining or recovery paths.
Triage one DeviceCloud run in a single call. Resolves the upload, folds retries per flow, and returns the failed flows with fail reasons, durations, and failure-screenshot paths (auto-downloaded from the HTML report), plus a passed/failed/flaky summary and suggested next steps. The highest-signal tool for debugging a red CI run — use this first, then read the screenshots and fix the flow or app code.
Classify every Maestro flow over a lookback window into healthy, flaky, broken, or regression, ranked worst-first, so you can tell whether a failure is worth fixing before diving in. Regressions (a flow that was passing and recently started failing) are surfaced first, since a recent change likely broke them. Use this to prioritize, then diagnose_run to fix a specific run. Filters mirror list_flow_analytics.
Output schemas not documented. Tools return ToolResponse with generic text content, but the actual JSON structure returned by DeviceCloud API (StatusJson, ResultsJson, FlowsJson types exist but not shown in schema) is not exposed to the LLM. LLMs cannot plan downstream operations without knowing what fields to extract.
No tool annotations. Both tools are read-only (Risk: READ_ONLY noted in metadata) but inputSchema lacks readOnlyHint annotation. LLMs cannot distinguish safe tools from destructive ones without explicit hints.
Error handling is generic. When uploadId/name lookup fails or API returns 400+, the tool returns 'HTTP {status}\n{body}' with no recovery guidance. Per pattern:recovery-guide, errors should suggest next steps (e.g., 'Try searching by name instead').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Parameter relationships undocumented. diagnose_run accepts uploadId OR name (mutually exclusive), but the description does not explicitly state this. LLMs may pass both or neither without clear guidance.
No pagination or result limits documented. suite_health filters by platform, appId, days, etc., but the response structure (how many flows returned, whether paginated, total count) is not visible. Large result sets could exhaust context.