Loading ToolBench
Multi-kit orchestration system for TDD, research experiments, and mathematics formalization with phase-based enforcement hooks and dashboard
Orchestration-kit defines 11 tools with complete JSON schemas and enums for constrained inputs (kit types, modes, priorities). However, descriptions are generic and lack actionable guidance. Most tool descriptions are 1-2 sentences without explaining WHEN to use each tool, what prerequisites exist, or how they chain together. Parameter descriptions are minimal (e.g., 'Action to run' for orchestrator.run's action param). No output schemas documented. Error handling is absent, no guidance on retryability, recovery steps, or what to do if a run fails. The pre-tool-use.sh hooks show sophisticated read guardrails and phase-specific enforcement, but these are not exposed as tool annotations or documented in the MCP interface.
Run full math cycle. Returns immediately with run_id. Poll kit.status or kit.runs for completion.
Run parallel batch of experiment RUN phases on separate EC2 instances. Returns immediately with run_id. Poll kit.status or kit.runs for completion.
Run research experiment cycle (frame/run/read/log) from a spec. Returns immediately with run_id.
Run full research cycle including survey (survey/frame/run/read/log). Returns immediately with run_id.
Run auto-advancing research program (picks next question from QUESTIONS.md). Returns immediately with run_id.
Run full TDD cycle (red/green/refactor/ship). Returns immediately with run_id. Poll kit.status or kit.runs for completion.
No output schemas documented. Tools return 'pointer-only paths' or 'run_id' but LLMs cannot plan downstream calls without knowing the structure of responses (fields, types, required vs optional).
Descriptions lack actionable guidance. Most are 1-2 sentences stating WHAT the tool does but not WHEN to use it, what prerequisites exist, or how it chains with other tools. E.g., 'Run a orchestration-kit action' does not explain when to call orchestrator.run vs kit.tdd.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Execute one request (by id or queue front) via tools/pump.
Return bounded log snippet via tools/query-log.
Create an interop request file via tools/kit request.
Run a orchestration-kit action via tools/kit and return pointer-only paths.
Return pointer summary for a run.
Parameter descriptions are minimal or missing context. E.g., 'Action to run' (orchestrator.run.action) does not explain what actions are valid, what format they take, or how they relate to the kit type. 'Reasoning' params lack guidance on expected length or content.
No error handling or recovery guidance. Tools return 'run_id' but do not document what happens if a run fails, how to check status, or what errors are retryable. Pre-tool-use.sh hooks enforce read budgets and phase constraints but these are not exposed as tool annotations or error messages.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) are absent. Orchestrator tools modify state (run, request_create, pump) but are not marked as destructive. Query and info tools are read-only but lack annotation. This prevents agents from reasoning about side effects and retry safety.