Loading ToolBench
Run, watch, and meter coding agents — Claude Code, OpenCode, Kiro, Codex CLI, Gemini CLI — with normalized events, cost accounting, A/B experiment/variant comparison, external run sources, and replayable run artifacts.
Server provides 10 tools with complete JSON Schema input definitions and descriptions. Naming follows verb_noun convention (harness_report, harness_run, harness_emit). Descriptions are present and substantive (avg ~120 chars), exceeding the 10-char minimum. However, parameter descriptions are inconsistent: some params lack detail (e.g., 'kiro' object in harness_run has only 'Kiro-specific run configuration' with no field breakdown). Output schemas are not documented in the tool definitions, callers must infer structure from implementation. Error handling is minimal; no recovery guidance or actionable error messages visible. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications (WRITE, READ_ONLY, REVERSIBLE). Parameter constraints are present (enums for agent/format) but some numeric params (budgetUsd, wallMs, idleMs) lack min/max bounds. Overall solid foundation but lacks polish expected of production-grade tools.
List known harness agents with their backing CLI command, whether the binary is resolvable on PATH, and the adapter's capability flags.
Emit agent run events in multiple formats (ATIF, OpenTelemetry, Langfuse) from a run file.
Preflight a Kiro run: verify the kiro-cli binary, auth, the native agent, that the model is offered and that session/set_model is acknowledged, plus MCP startup notices — over a real ACP handshake that sends NO prompt (no tokens spent). Returns a receipt of verified/failed/unproven checks.
Render a self-contained HTML comparison report for a trial directory (same as `harness report`).
Run one harness agent (claude|opencode|kiro|codex|gemini) with a prompt and optional model/cwd/budget/turn limits; resolves with the full RunResult (sessionId, events, tokens, totalCost, durationMs, exitStatus, warnings).
Start a harness agent run (claude|opencode|kiro|codex|gemini) WITHOUT waiting for completion; returns { runId, sessionId: null, started, transcriptPath } promptly. Poll harness_run_status, page harness_run_events, or harness_run_cancel with the runId.
Output schemas not documented. Tool definitions lack return type specifications. Callers must reverse-engineer response structure from implementation or runtime behavior.
Numeric parameters lack bounds. budgetUsd, wallMs, idleMs, cursor, limit, sinceDays have no min/max constraints. LLMs may pass absurd values (negative durations, billion-item limits).
Complex object parameters lack field documentation. 'kiro' object in harness_run and harness_run_async has only generic description; no breakdown of expected fields, types, or constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
Cancel an in-flight async harness run by runId. Returns { cancelled: true } if the run was found and aborted, or { cancelled: false } if the run was not found or already completed.
Page through the raw transcript events of an async harness run. cursor is a line offset (default 0); limit is events per page (default 100, max 1000). Returns { events, nextCursor, total, truncated }.
Report the status of an async harness run: effectiveStatus (running/interrupted/success/error/aborted), sessionId, token/cost totals, last event, timestamps, and elapsed time.
Report usage statistics (runs, tokens, cost) for the past N days, optionally filtered by agent.
No tool annotations despite clear risk classifications. Tools marked WRITE/REVERSIBLE lack destructiveHint or idempotentHint annotations. Agents cannot infer safety properties from schema alone.
Error handling not visible in tool definitions. No recovery guidance, error classification, or actionable error messages documented. Agents will not know whether to retry, ask user, or abort.