Loading ToolBench
Relay - AI agent latency auditor as an MCP server and CLI (an Arclat product). Audits Python codebases for latency bottlenecks in AI agents using static analysis and semantic verification.
Relay has 4 tools with clear action verbs and reasonable descriptions (avg 180 chars). All tools have input schemas with type definitions. However, parameter descriptions are sparse or missing entirely, most params lack guidance on format, constraints, or valid values. Output schemas are not documented. Error handling is minimal; no recovery guidance or actionable error messages visible. The server maintains state in ~/.relay/last-audit.json, which is unusual for MCP but functional. Tool composition is sound (each does one thing), but parameter constraints (enums, ranges) are absent.
Apply a fix recommendation from the last audit to a file.
Audit an AI agent codebase for latency bottlenecks. Reads Python files in the given directory and returns ranked findings with fix recommendations. Every finding also carries a verification verdict (confirmed / dismissed with a reason) from a semantic second pass.
Return findings from the last audit, optionally filtered.
Show the verification verdicts for the last audit: which findings the semantic pass confirmed as real latency gaps and which it dismissed as likely false positives (with the reason for each).
Parameter descriptions lack format constraints and valid value guidance. 'severity' in get_findings says 'Filter by severity - high, medium, or low' but does not declare an enum. 'finding_type' and 'status' similarly lack formal constraints. LLMs will hallucinate invalid values.
Output schemas are not documented. Tools return formatted markdown strings (e.g. _format_findings) or JSON objects, but the response structure is not declared. LLMs cannot plan downstream calls or extract structured data reliably.
Error handling is absent. No visible try-catch blocks, validation, or recovery guidance in tool implementations. If audit_codebase fails (bad directory, permission denied, AST parse error), the LLM receives no actionable error message.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
apply_fix supports dry_run but lacks confirmation/approval pattern for destructive operations. An LLM could set dry_run=False and modify source files without explicit user consent, risking unintended code changes.
State persistence via ~/.relay/last-audit.json is stateful and session-dependent. MCP protocol is stateless; each request should be self-contained. If the server restarts or runs in a different environment, state is lost or inaccessible.