Loading ToolBench
An evidence-backed factory that mines workflows, tests improvements, and queues proven winners for human approval.
Critical gaps across all dimensions. Tool descriptions are verbose (500-1000+ chars) and domain-specific jargon-heavy, making them difficult for LLMs to parse for selection. Input schemas are present but lack parameter-level descriptions for most fields. No output schemas documented. Error handling is absent, tools do not guide recovery or categorize failures. Tool names lack clear verb-noun structure (e.g., 'loop_start' vs 'start_loop', 'request_next_phase' vs 'get_next_phase'). Security concerns: no evidence of secret injection, permission gates, or audit trails. Composition is poor, tools are tightly coupled to internal state (runId, phase gates) rather than composable primitives. This server is domain-specific research infrastructure, not a general-purpose agent tool suite.
Persist a raw artifact (run log, baseline copy) with a sha256 hash. role:"baseline" hash-locks the baseline (write-once; tampering refused). Pass measurement:{tokenCost,quality} so the artifact can serve as a tool-measured, reverifiable measurementRef. sourcePath reads are disabled; pass explicit content.
Bench-maker only: freeze a benchmark directly (benchSource:maker) without worker benchmark_propose. Defaults benchPartition to gate (held-out). Worker benchmark_propose becomes a no-op while this scorecard is frozen.
Propose one or more benchmark scorecards built from real prior uses/failures. Each needs ≥1 task-value dimension, ≥1 resource/cost dimension, and ≥1 concrete case, or it is rejected as a hand-waved benchmark.
Record a tool-measured run of an arm through the frozen benchmark. arm:"baseline" sets the bar challengers must beat. Requires a measurementRef → a recorded raw artifact; model self-report never sets the bar.
Freeze ONE proposed benchmark as the immutable scorecard for this cycle. Requires the baseline to be hash-locked first. Changing a frozen benchmark needs a new epoch + rationale.
Tool descriptions are 500-1000+ characters of domain jargon (e.g., 'Ask-once gate', 'phase-gated streaming', 'hash-lock baseline'). LLMs cannot parse these for tool selection. Baseline: 194 chars avg, 10-1024 range. These descriptions violate the 'too long' constraint and bury actionable intent.
No parameter-level descriptions. Schema shows property names (e.g., 'runId', 'loop', 'phase') but lacks descriptions explaining what each controls. LLMs cannot infer meaning from names alone. Baseline: 100% of A+ tools have param descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 29 | 2026-07-28+ | v2 |
SUPERVISOR-EXECUTED full test (off by default; opt in with env SUPER_LOOP_ALLOW_EXEC=1). Loop Factory itself LAUNCHES 3-5 allowlisted frontier workers (claude/codex/glm/gemini binaries on PATH). Native executables run directly without a shell; an allowlisted Windows .cmd/.bat shim alone uses the guarded cmd.exe adapter. Prompts remain on stdin. Capture stdout/stderr and measure token cost + quality from tool output.
Ask-once gate. Confirms the task before any loop runs. If the task is underspecified, returns one brief explanation plus a few short questions once (goal; PATH — improve / discover / mine; loop or domain; corpus scope + order; what "better" means; hard limit; ONE model-choice question — primary/test/builder/judge, or press enter for defaults, or say "any model" to disable the banlist; and a deeper-explanation offer); call again with { answers } to begin. Current defaults: GPT-5.6 Sol primary/test work, Fable 5 plus GPT-5.6 Sol build/gating, Fable 5 judge. The supervisor still owns measurement, integrity, and promotion. Stores every user message locally with a sha256 hash. After initialization it does not ask again or mark the campaign complete; the operator remains the stop condition and the dashboard stays available.
Alias of request_next_phase.
Begin phase-gated streaming of a bundled or custom local loop. Opens/activates the supervisor lane for that loop. Use "strip-miner" (The Strip Miner Loop / cross-agent source miner, 345 lines), "loop-de-loop" (Loop 2, the improvement loop, 75 lines), or any id registered with loop_register. Returns ONLY section 0; the full loop stays inside the supervisor.
Record lightweight evidence for the current phase (what you actually did/observed). Attach { loop, phase } to satisfy the phase gate and unlock the next section.
Register 3–5 challenger hypotheses, each on a route allowed by the run's modelPolicy banlist (defaults reject haiku/mini/nano/lite/prior-gen). Requires baseline hash-lock + frozen benchmark + measured baseline bar (benchmark-first). Rejects <3 or >5.
Stream the next loop section. BLOCKED (PHASE_SKIP) unless the current section already has recorded evidence. Prevents 300+ lines collapsing into the model before real decisions.
Record ONE full test of a hypothesis = 3–5 frontier agents that actually ran the loop end-to-end. Every agent run must carry a measurementRef (tool-measured). Aggregates vs the frozen baseline bar; a no-improvement run is NO_IMPROVEMENT, never "perfect", and bumps the failure counter.
No output schemas documented. Tools return results but LLMs have no specification of return fields, types, or structure. Baseline: 100% of A+ tools document return types. This forces LLMs to guess downstream field names and types.
No error handling guidance. Tools do not specify error conditions, recovery steps, or error categorization (retryable vs fatal). Baseline: A+ tools include recovery guides and error classification. Agents cannot self-correct on failures.
Tool names lack clear verb-noun structure. 'loop_start' (start what?), 'request_next_phase' (request from whom?), 'loop_next' (alias of previous, redundant). Baseline: 90% of A+ tools use verb_noun (get_, create_, update_, search_). Ambiguous names cause LLM selection errors.