Loading ToolBench
Test orchestration framework that exposes test execution capabilities to AI agents via MCP (Model Context Protocol) Server
Four tools with basic descriptions and input schemas, but significant gaps in parameter documentation and output schema clarity. Tool names follow verb_noun convention (taichi_run, taichi_list, taichi_failures, taichi_regression), which is positive. However, parameter descriptions are minimal or absent for most inputs. No output schemas documented. Descriptions are adequate length (50-100 chars) but lack actionable context for LLM selection. No error handling guidance. No parameter constraints (enums, ranges, formats). Missing idempotency and retry semantics for orchestration tools.
Retrieve failing test cases from the most recent test run reports and provide failure context for analysis
List all projects, environments, skills, and report configuration from the taichi configuration file
Run regression tests on a specific project to verify fixes and prevent regressions with optional timeout
Execute test orchestration according to configuration, optionally filtering by project and skills with timeout support
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract structured data without knowing return types.
Parameter descriptions are missing or trivial. 'config_path' and 'reports_dir' lack format/constraint guidance. 'timeout' lacks format examples (is '30s' or '30' expected?).
No error handling guidance. Tools do not indicate what errors are retryable, what user actions are needed, or how to recover from failures.
No parameter constraints. 'timeout' accepts free-form strings without validation rules. 'skills' array lacks item constraints or examples.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
Idempotency and retry semantics not documented. Orchestration tools (taichi_run, taichi_regression) should declare whether repeated calls with same inputs produce same results.