Loading ToolBench
Domain-agnostic MCP server for autonomous experimentation. Generalises the autoresearch pattern (modify → run → measure → keep/discard → repeat) into a reusable server that any AI agent can drive via MCP tools.
Strong foundation with 11 well-named tools following verb_noun convention (autoexp_get_*, autoexp_update_*, etc.). All tools have descriptions (avg ~120 chars, within baseline 34-392 range). Input schemas present with type definitions and parameter descriptions. Tool annotations (readOnlyHint, destructiveHint, idempotentHint) correctly applied. Key gaps: output schemas not formally documented in code; error handling lacks recovery guidance; some descriptions could be more prescriptive about when/why to use each tool.
Start an experiment lifecycle and persist a pending history entry. Use this when experiment execution happens outside autoexp_run_experiment (for example when another MCP server runs the simulation).
Complete a pending experiment with metric and optional metadata. Use this when the experiment was started with autoexp_begin_experiment and execution happened externally.
Discard a pending experiment without running it.
Return the most recent experiments with their metrics and outcomes. Use this to review what has already been tried, spot patterns, and avoid repeating failed approaches.
Return the current session status: project info, best score, total experiments run, and which files are editable. Use this at the start of a session to orient yourself, or between experiments to check progress.
Find-and-replace within an editable file. The old_text must appear exactly once in the file. This is safer than a full file replace for small, targeted modifications.
Output schemas not formally documented. Tools return JSON strings but no schema definition visible in code for downstream tool chaining. Agents cannot predict response structure for planning.
Error handling lacks recovery guidance. ToolError exceptions raised (e.g., 'File not found', 'not editable') do not suggest next steps. LLM cannot self-correct without explicit guidance.
Descriptions for experiment lifecycle tools (begin, run, complete, discard) lack clarity on when to use each vs. alternatives. 'Use this when...' guidance is minimal, forcing LLM to infer intent.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Read the contents of any file declared in the config (editable or read-only). Use this to understand the current state of the code/config before planning your next experiment.
Restore editable files from a saved snapshot (begin or final phase of a past experiment).
Execute an experiment: run the command, extract the metric, evaluate keep decision, and persist the result.
Mark an existing completed experiment as the baseline.
Overwrite an editable file with new content. Use this when you want to replace the entire file. For smaller, targeted changes, prefer autoexp_patch_file instead.
No pagination or result limits documented for get_history. Returning 200 experiments could exhaust context window. Baseline rubric requires pagination guidance.
Destructive operations (update_file, patch_file, run_experiment) lack confirmation or dry-run pattern. Agents can irreversibly modify files without safeguards.