A single-process Slack bot using Socket Mode that runs AI agents for PR review, code fixes, and question answering with configurable models and execution environments
Switchboard provides 9 tools with basic schemas and descriptions, but falls short of production quality. Tool names follow verb_noun convention (bash, read, grep, find, ls, write, edit, submit_pr_description, submit_verdict), which is good. However, descriptions are minimal (10-50 chars), parameter descriptions are sparse or missing, and output schemas are not documented. The edit tool's 'edits' parameter lacks type detail. No error handling guidance, no enum constraints on verdict/status fields, and no indication of idempotency or retry safety. Tools like submit_pr_description and submit_verdict are write operations but lack confirmation patterns or dry-run support. Parameter naming is inconsistent: 'path' vs 'pattern' for similar operations. No documentation of which tools are idempotent or safe to retry.
Execute shell commands in the checkout directory with configurable timeout and environment
Edit files in the checkout directory with line-based modifications
Find files in the checkout directory matching criteria
Search for patterns in files within the checkout
List directory contents in the checkout
Read file contents from the checkout directory
Submit or update a pull request description
Output schemas not documented. LLMs cannot plan downstream tool calls or extract structured data from results. No indication of what fields bash, read, grep, find, ls return.
Parameter descriptions are minimal or missing. 'pattern' in grep/find lacks explanation of regex vs glob syntax. 'edits' in edit tool has no type detail (array of what?). LLMs cannot infer parameter semantics.
No error handling guidance. Tools like bash and write can fail in many ways (permission denied, timeout, disk full) but provide no recovery hints. LLMs cannot self-correct.
Destructive operations (write, edit, bash with rm, submit_verdict) lack confirmation or dry-run patterns. Agents can accidentally delete files or submit wrong verdicts without safeguards.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Submit a verdict or review decision on a pull request
Write or create files in the checkout directory
submit_verdict 'verdict' parameter has no enum constraint. LLMs may hallucinate invalid verdict values. No documentation of valid options (approve, request_changes, comment, etc.).