Loading ToolBench
MCP server exposing a running atomics API as tools for evaluating and benchmarking language models. Provides asynchronous job submission and polling for runs, evals, sweeps, stress tests, and soak tests.
Server has 15 tools with complete input schemas and descriptions. Naming follows verb_noun convention (list_, submit_, get_, compare). Descriptions are detailed (100-300 chars typical) and explain WHEN to use each tool. However, output schemas are not documented, LLMs cannot predict response structure for planning. Parameter descriptions are present but some lack constraints (e.g., 'effort' accepts 7+ values but no enum). Error handling guidance is minimal. No tool annotations (readOnlyHint/destructiveHint) despite clear risk distinctions (READ_ONLY vs WRITE). Missing idempotency guidance for retry-heavy workflows.
Compare two runs side-by-side.
Poll a job for its current status, progress, and results. Returns job details including request, progress, and result payload.
Retrieve a saved benchmark run by ID.
Check that the atomics API server is reachable and serving. Call this first when another tool reports a connection problem.
List all in-memory API jobs. The result payload is omitted; poll `/jobs/{id}` for details.
List models loaded on an Ollama or vLLM instance. `provider` is `ollama` or `vllm`. Pass `host` to override the server's configured endpoint. Does not generate and does not spend tokens.
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination) for planning downstream calls or extracting data. submit_eval, submit_sweep, submit_battery, get_job responses are opaque.
Enum constraints missing for multi-value parameters. 'effort' accepts 7+ values (none/minimal/low/medium/high/xhigh/max/xl/ultra) but no enum in schema, LLMs may hallucinate invalid values. 'suite' and 'channel' similarly lack enums.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint). Tools are marked READ_ONLY vs WRITE in metadata but not in schema. LLMs cannot infer retry safety or side-effect scope without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
Health-check a provider and generate a fixed 2+2 probe. Spends a few tokens. The prompt is fixed server-side. Use this after `list_models` to confirm a tag answers before submitting an eval. `effort` is the shared reasoning dial (none/minimal/low/medium/high/xhigh/max; aliases `xl` and `ultra`). `reasoning_mode` is OpenAI-only (`standard`/`pro`).
List recent benchmark runs.
Run a named battery as one job and return its job id. Spends tokens. `budget_usd` is required. `name` is one of the five batteries (`desk-pass`, `blue-capability`, `red-capability`, `agent-gate`, `threat-model`). provider-test runs the fixed 2+2 probe; qa runs the fixture file or profile and a FAIL marks the step not-ok. archreview stays on the CLI. `thinking` / `effort` forward onto every step (battery default is `--no-thinking` when `thinking` is unset). `host` is the inference endpoint; `judge_host` is the judge endpoint when it differs. Poll `get_job` until `status` is `completed`.
Start an eval suite and return its job id immediately. Spends provider tokens, for the judge as well as the model under test. `suite` is one of `accuracy`, `rag`, `multiturn`, `adversarial`, `codegen`, `refusal`, `redblue`, `toolcall`, or `codereview`. `budget_usd` caps the combined spend; leave it unset to accept the server's default ceiling. `host` is the inference endpoint (same meaning as `list_models`). `judge_host` is the judge endpoint when it differs from `host`. `runs` is 1–3 on `adversarial`, `redblue`, and `toolcall` only. `channel` is `both`, `tools`, or `prose` on `toolcall` only (CLI `--channel`). `effort` / `reasoning_mode` are the shared reasoning dial (same values as `provider_test`). Poll `get_job` for live `request` / `progress` and growing `result.fixtures` (every suite).
Start a benchmark run and return its job id immediately. Spends provider tokens. `provider` is a provider name such as `claude`, `openai`, or `ollama`; omit `model` to use that provider's default. `tier` selects the task difficulty profile. `effort` / `reasoning_mode` are the shared reasoning dial (same values as `provider_test`). `host` is the inference endpoint (same meaning as `submit_eval`). Poll `get_job` for the result.
Hold concurrency and classify drift. Duration is seconds, max 300.
Ramp concurrency to find saturation. One model, required budget, capped at 8 concurrent and 15s per phase.
Start a multi-model, multi-suite campaign and return a job id. Spends tokens. `budget_usd` is required. Name the models — there is no discover-everything flag; call `list_models` first. `suites` is a list from `eval`, `redblue`, `refusal`, `toolcall`, `codereview`. `runs` is 1–3. `host` is the inference endpoint (same meaning as `submit_eval`). `judge_host` is the judge endpoint when it differs from `host`. Poll `get_job` until `status` is `completed`.
Analyze trends in benchmark results over time.
Error handling guidance absent. Tools like submit_eval and submit_sweep spend tokens and return job IDs, but no description of failure modes (budget exceeded, invalid suite, network timeout) or recovery steps.
Parameter constraints underspecified. 'iterations' (submit_run), 'duration' (submit_soak), 'limit' (recent_runs) lack min/max bounds. LLMs may pass absurd values (iterations=1000, duration=999999).