Loading ToolBench
Self-reflective feedback for agent harnesses, derived from OTEL trace analysis. An MCP server that exposes the pipeline's stage-6 execution graph as a queryable relational surface, so an analyst agent can drive its own analyses (read-only SQL + graph traversal) instead of consuming fixed findings.
Seven tools with strong descriptions (194 - 350 chars, well above baseline 194 avg) and clear semantic purpose. All tools have explicit input schemas with Zod validation. Naming is verb-forward (load_, describe_, query, resolve, subtree, causal_path, open_viz). However, parameter descriptions lack detail on constraints, ranges, and error recovery. Output schemas are documented in code but not exposed to the LLM in tool definitions. Error handling is minimal, no recovery guidance or categorization. Tools are read-only (low risk) but lack idempotency hints and confirmation patterns for the viz server startup.
Walk the causal DAG from a node id: `upstream` for the causes that triggered it (default), `downstream` for what it triggered.
Return the queryable schema: tables, columns (with docs), the semantic vocabulary, and example queries (the common rollups as SQL). Works without a dataset loaded — call it first to plan your queries.
Load a dataset and make it queryable, replacing any previously loaded one. Pass `repo` (a repo name like 'coach', or an absolute repo path) to load that repo's Claude Code logs across the main checkout AND every git worktree by default — the harness-agnostic way to get a complete picture of a repo. Or pass `path` for an exact directory of OTEL Tempo traces / native session .jsonl logs. Either way it runs through the full pipeline and is materialized. Returns a summary (the directories loaded + counts). Call this before querying unless a dataset was preloaded at startup.
Open the interactive graph visualization. Starts a local web server over the built app and the stage JSON dumped into the `out/` dir by the last directory `load_dataset`, and returns a URL. Pass a dumped JSON file name (default `07-resolved-graph.json`). Use `focus` to center on a single node, OR `source` + `dest` to highlight a related PAIR distinctly (e.g. a failed call as `source` and the call that recovered it as `dest`) and fit both into view at once — either of source/dest alone is also valid. Requires the app to be built (`pnpm --filter @coach/app build`).
Parameter descriptions lack constraint details. 'id' param in resolve/subtree/causal_path has no format, length, or example. 'direction' enum in causal_path is documented but others lack enum clarity.
Output schemas not exposed in tool definitions. Tool descriptions mention 'truncated', 'droppedRows', 'notice' fields but LLM cannot see the schema structure. Agents must infer output shape from description text.
No error recovery guidance. 'query' mentions truncation and caps but does not explain how to recover (e.g., 'add LIMIT clause', 'use pagination'). No actionable error messages for invalid SQL or missing datasets.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Run a read-only SQL query (a single SELECT or WITH statement) over the loaded execution-graph tables. Results are capped (≤1000 rows and a serialized-byte budget). When anything is reduced, `truncated` is true, `droppedRows` reports how many rows were cut (`returnedRows` is what you got of `rowCount` total), and `notice` explains in plain language which cap fired and how to recover — read it. See describe_schema for the table/column reference.
Resolve one node id across all three layers (node data, message deltas, semantic label) — the full hydrated node.
Return the containment descendants of a node id (what ran within its time span), via the containment relation.
load_dataset parameters 'repo' and 'path' are mutually exclusive but not documented. LLM may pass both or neither, causing ambiguity. Missing guidance on when to use each.
open_viz starts a local web server with side effects but lacks confirmation/dry-run pattern. No warning that this modifies local state or blocks until server stops. Agents may invoke unexpectedly.