Deployment agent that rehearses, ships, and observes — without touching your code. Exposes the deployment pipeline as agent-callable tools for planning, applying, watching, approving, and diagnosing deployments.
Convoy has 8 well-named tools with clear action verbs (convoy_plan, convoy_apply, convoy_status, etc.). All tools have descriptions (avg ~150 chars, within baseline 194). Input schemas are present and mostly typed. However, several critical gaps reduce the score: (1) Parameter descriptions are sparse or missing for nested objects (e.g., byok.provider, byok.apiKey lack descriptions in the schema view); (2) Output schemas are not documented, callers cannot see what convoy_plan, convoy_apply, or convoy_status return; (3) Error handling is minimal, no recovery guidance, no categorization of retryable vs fatal errors; (4) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels (WRITE, IRREVERSIBLE, READ_ONLY); (5) Secrets (ANTHROPIC_API_KEY, SSH private keys) are exposed as parameters, violating pattern:secret-injection. The code shows good intent (tool registration, structured input), but lacks production-grade polish.
Apply a saved plan: spawn the orchestrator, materialize the run, and stream events back to the caller. Returns immediately with a runId; the caller watches via convoy_status or the web UI.
Approve or reject a pending approval (e.g. canary promotion, rollback decision). Caller supplies the approval ID from convoy_status.
Diagnose a failed run: analyze logs, events, and state to suggest fixes. Returns a diagnosis with root-cause analysis and remediation steps.
List saved deployment plans (newest first). Use a returned planId with convoy_apply.
List recent runs (newest first). Each run shows status, platform, and completion time. Use a returned runId with convoy_status or convoy_diagnose.
Scan a local repository and create a deployment plan (platform pick, drafted files, rehearsal + rollback strategy). Saves the plan and returns its planId. Next step: convoy_apply with that planId.
Output schemas not documented. Tools like convoy_plan, convoy_apply, convoy_status, convoy_diagnose return complex objects (planId, runId, events, approvals, diagnosis) but callers cannot see the structure. LLMs cannot plan downstream calls or extract required fields.
Secrets exposed as parameters. convoy_plan and convoy_apply accept ANTHROPIC_API_KEY (via byok.apiKey) and convoy_vps_bootstrap accepts SSH private keys as plaintext parameters. These will be logged in agent traces and prompt history, violating pattern:secret-injection.
No tool annotations. Tools marked WRITE, IRREVERSIBLE, and READ_ONLY in metadata but not exposed via readOnlyHint/destructiveHint/idempotentHint in schema. Agents cannot infer safety properties from the tool definition alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Poll a run's status and event stream. Returns the run summary, all events so far, and any pending approvals. Safe to call repeatedly; events are immutable once recorded.
Bootstrap a VPS target: provision SSH access, install Docker, and validate connectivity. Returns the target ID and live URL for monitoring.
Nested parameter descriptions missing. byok object has properties (provider, apiKey, siteUrl, clientId, clientSecret, projectId, environment, secretName) but most lack descriptions in the schema. LLMs cannot determine which fields are required or what they control.
No error handling guidance. Tools return success/failure but do not categorize errors as retryable, user-fixable, or fatal. No recovery hints (e.g., 'Plan not found. Call convoy_list_plans to see available plans'). Agents cannot self-correct on failure.