Loading ToolBench
MCP server that exposes evalview check/snapshot as MCP tools for Claude Code. Provides test case creation, regression checking, baseline snapshotting, and skill validation for AI agent testing.
EvalView MCP server has 8 tools with generally clear naming (verb-first: create_, run_, list_, validate_, generate_, etc.) and substantive descriptions (100-300 chars each). However, input schemas are partially visible but lack complete type definitions for all parameters. Several tools have descriptions that reference auto-detection logic but do not formally specify parameter constraints, enums, or validation rules. Error handling guidance is minimal, tools do not document recovery paths or categorize failures. Output schemas are not documented in the tool definitions. The server addresses a specialized domain (agent testing/evaluation) with reasonable composition, but falls short of production-grade rigor in schema completeness and error guidance.
Create a new EvalView test case YAML file for an agent. Call this when the user asks to add a test, or when you want to capture expected agent behavior. After creating a test, call run_snapshot to establish the baseline. No YAML knowledge required — just describe the test. IMPORTANT: Automatically detect test_path by looking for a 'tests/evalview/' directory in the current project. If found, use it. Otherwise use 'tests'.
Auto-generate test cases from a SKILL.md file. Call this when the user asks to create tests for a skill — it reads the skill definition and generates a ready-to-run YAML test suite covering explicit, implicit, contextual, and negative test categories. After generating, call run_skill_test to execute them.
Generate a beautiful self-contained HTML visual report from the latest evalview check or run results. Opens automatically in the browser. Call this after run_check or run_snapshot to give the user a visual breakdown of traces, diffs, scores, and timelines. Returns the absolute path to the generated HTML file.
List all available golden baselines in this EvalView project. Shows test names, variant counts, and when each baseline was last updated.
Check for regressions against the golden baseline. Returns diff output showing what changed vs the last snapshot. A regression means the agent's behavior changed unexpectedly. Use this after refactoring agent code to confirm nothing broke. IMPORTANT: Automatically detect test_path by looking for a 'tests/evalview/' directory in the current project. If it exists, pass it as test_path. If the project has a custom test location, use that instead.
Output schemas not documented. Tools return results but descriptions do not specify what fields/structure the LLM should expect. This forces LLMs to guess at response shape and risks failed downstream chaining.
Parameter constraints not formalized. 'min_score' in create_test is described as '0-100' in prose but lacks JSON Schema min/max. 'count' in generate_skill_tests lacks bounds. Enums (e.g. agent types in run_skill_test) are listed in description text, not as formal enum constraints.
Error handling guidance absent. Tools do not document what errors can occur, how to recover, or what the LLM should do next. E.g., run_check does not explain what happens if test_path is invalid or if no tests exist.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Run a skill test suite against a SKILL.md. Executes two evaluation phases: Phase 1 (deterministic) checks tool calls, file operations, commands run, output content, and token budgets. Phase 2 (rubric) uses LLM-as-judge to score output quality against a defined rubric. Call this after writing skill tests or after any change to the skill or agent. Use --no-rubric for fast Phase 1-only checks with no LLM cost.
Run tests and save passing results as the new golden baseline. Use this to establish or update the expected behavior after an intentional change. Future `run_check` calls will compare against this snapshot. IMPORTANT: Automatically detect test_path by looking for a 'tests/evalview/' directory in the current project. If it exists, pass it as test_path.
Validate a SKILL.md file for correct structure, naming conventions, and completeness. Call this after writing or editing a SKILL.md before running tests. Returns a list of issues found and whether the skill is valid.
Auto-detection logic in descriptions is procedural, not declarative. 'Automatically detect test_path by looking for...' describes implementation detail rather than tool contract. If auto-detection fails, the tool should accept test_path as a fallback, but this is not clearly stated.
Idempotency not declared. run_snapshot and run_skill_test modify state (save baselines, generate tests). Descriptions do not clarify whether repeated calls with identical inputs produce identical results or accumulate changes.