Loading ToolBench
DeployBot exposes 22 tools via FastMCP with consistent naming (verb_noun pattern) and descriptions present for all tools. However, critical gaps undermine production readiness: (1) Input schemas are inferred from Python type hints but NOT explicitly visible in JSON Schema format in the source, cannot verify parameter types, constraints, or enums; (2) Parameter descriptions are minimal (5-15 chars) and lack actionable detail, e.g., 'Pull request identifier' does not specify format (owner/repo#123 vs just 123); (3) Output schemas are completely undocumented, tools return JSON strings but structure is opaque to LLMs; (4) No error handling guidance, RuntimeError on CLI failure gives no recovery path; (5) No pagination, limits, or result-shaping for list-like tools (queue_plan, pipeline_status, delivery_metrics); (6) Destructive tools (drain_queue, react_to_delivery_event) lack confirmation or dry-run patterns. Tool names are clear (enqueue_pull_request, freeze_queue) but parameter design forces LLMs to guess formats.
Acknowledge only after the deployed message reaches the native thread.
Block a queued PR with a concrete reason while other work continues.
Cancel one durable deploy request before it merges.
Elect one native thread to repair the current failed main release.
Scaffold a cumulative PR for overlaps or full-batch validation.
Read p50/p95 delivery timing for recent merged pull requests.
Revoke merge authorization for one queued PR.
Input schemas not visible in source code. Python type hints exist but no explicit JSON Schema definitions provided to FastMCP. Cannot verify parameter constraints, enums, or formats. All 22 tools affected.
Parameter descriptions are minimal (5-15 chars) and lack actionable detail. 'Pull request identifier' does not specify format (owner/repo#123, #123, or full URL). 'Configuration file path (optional)' does not explain where to find defaults or what format is expected.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
Run read-only installation, policy, and GitHub health checks.
Merge every independent ready PR in the frozen batch.
Queue one exact reviewed PR head after the user authorizes deploy.
Advance the newest exact-main release through verified deployment.
Freeze the current queue membership and exact head SHAs.
Evaluate one exact PR head without granting merge authorization.
Read active threads, PR stages, queue, CI, and deployment status.
Promote every exact-head-ready deploy request into the merge queue.
Read the ordered queue, blockers, and source-overlap groups.
Run the event-driven promotion, batching, merge, and optional release flow.
Bind existing user intent to a freshly reviewed replacement head.
Persist intent and route receipts to the recorded PR-opening thread.
Atomically verify, unblock, requeue, and wake a repaired pull request.
Clear a resolved queue blocker.
Publish state; opening PR phases immutably bind the native owner.
Output schemas completely undocumented. All tools return JSON strings (via --json flag) but structure is opaque. LLMs cannot plan downstream calls or extract fields without trial-and-error parsing.
Destructive/irreversible tools (drain_queue, react_to_delivery_event) lack confirmation or dry-run patterns. Agents can merge all PRs or trigger releases without explicit user approval, risking production incidents.
Error handling provides no recovery guidance. RuntimeError on CLI failure returns stderr/stdout as-is. LLMs receive raw error messages with no actionable next steps (e.g., 'Try search_users() first' or 'Retryable: rate limit exceeded').