Loading ToolBench
Diagnose your AI agents in production — extract behavioral policies from prompts, evaluate traces against them, and generate diagnostic reports.
Strong tool definitions with comprehensive parameter schemas and clear descriptions. All 11 tools have explicit registrations with input schemas and descriptions. Tool names follow verb_noun convention (triage_init, triage_explain, triage_analyze). Descriptions are detailed (100-300 chars) and explain WHAT, WHEN, and cost implications. Parameters are well-typed with Zod schemas and descriptions. However, output schemas are not documented in the tool definitions, LLMs cannot see what fields to expect from responses. Error handling is present but generic (try/catch with error message). Some parameter descriptions could be more prescriptive about constraints and formats.
Full evaluation: run metrics + policy compliance checks + diagnosis + fix generation on all conversations. Generates report.json and report.html. High LLM cost. Returns summary of compliance, worst policies, and recommendations.
Targeted policy compliance check on specific conversations — evaluates only policy violations without computing metrics. Moderate LLM cost. Faster than triage_analyze. Returns pass/fail verdicts per policy.
Run a demo with built-in example agents and traces. Generates a full report with sample data. High LLM cost. Useful for testing the workflow without real production data.
Compare two report.json files to see what changed. Zero LLM cost. Returns policy compliance deltas, new/resolved failures, and improvement summary. Use this to verify that a prompt fix actually improved agent behavior.
Deep-dive diagnosis of a single conversation — root cause, cascade chain, blast radius, and suggested fix. If a report.json exists with a diagnosis, returns it at zero cost. Otherwise generates a diagnosis (moderate LLM cost). Use worst=true to automatically pick the worst conversation from the last report.
Output schemas not documented in tool definitions. LLMs cannot see what fields to expect from responses (e.g., triage_analyze returns report.json structure, triage_sample returns conversation list). This forces LLMs to guess downstream field names and risks broken tool chains.
Multiple trace source parameters (traces, langsmith, otel, axiom, langfuse) are mutually exclusive but not documented as such. LLMs may pass multiple conflicting sources, causing ambiguous behavior. Should state 'Choose ONE of: traces, langsmith, otel, axiom, or langfuse' in each parameter description.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 67 | <=2025-11-25 | v2 |
Show run history — compliance trends across analyze runs. Zero LLM cost. Reads .triage-history.jsonl and displays compliance over time. Use this to track agent health improvements across multiple evaluation runs.
Extract testable behavioral policies from an agent's system prompt using LLM analysis. Moderate LLM cost (~1 API call). Writes policies.json for use by other tools.
List all behavioral policies from policies.json. Zero LLM cost. Returns policy IDs, names, descriptions, categories, and complexity scores. Use this to map a problem description to specific policy IDs for targeted checking.
Fetch a small sample of recent conversations for inspection. Zero LLM cost. Provide a JSON traces file (recommended) or a LangSmith/OTel source. Use this to look at raw conversation data before running expensive analysis. Supports keyword search on message content to find conversations about specific topics. This is the key tool for narrowing from 'something is wrong' to 'these conversations show the problem'.
Quick health check from the last analysis report. Zero LLM cost — reads report.json from disk. Returns overall compliance, worst policies, critical issues, and a suggested next step. Use this first to understand current agent health before running expensive analysis.
Open the generated HTML report in your browser. Zero LLM cost. Launches report.html in the default browser for interactive exploration of results.
Error handling is generic (catch-all with error.message). No categorization of retryable vs fatal errors, no recovery guidance. E.g., 'File not found' should suggest 'Run triage_init first to generate policies.json' or 'Check the traces file path.'
Parameter descriptions lack format/constraint details. E.g., 'since' accepts '2h, 24h, 7d' but description doesn't specify the regex pattern or valid units. 'max_conversations' has no min/max bounds stated. LLMs may pass invalid values like '999999' or 'invalid_time'.
triage_view tool launches a browser (side effect) but description does not warn that this is a UI action with no return value. LLMs may expect structured output and retry on timeout. Should clarify: 'Opens report.html in default browser. Returns success/failure only, no structured data.'