Loading ToolBench
Scientific Research Reasoning Engine for evaluating experimental and empirical claims — auditing ML training runs, ablations, and sweeps; reviewing papers; checking reproducibility; and producing structured research reports. Optional Weights & Biases integration via the experiment-audit-mcp server.
8 tools with complete JSON schemas and descriptions present. Naming follows verb_noun pattern (test_connection, list_runs, get_run_summary, etc.). Descriptions range 80-180 chars, meeting the 10-1024 baseline. All parameters have type definitions and descriptions. However, descriptions lack actionable context (WHEN to use, prerequisites, dependencies between tools). No output schemas documented, LLMs cannot infer response structure for chaining. Error handling guidance absent. Parameters like 'backend' are repeated across all tools but lack enum constraints (only 'wandb' mentioned in descriptions, not formalized). No pagination guidance for list_runs despite potentially large result sets. Audit tools (audit_training_curve, audit_ablation, audit_sweep) lack detail on confidence scoring, return structure, and interpretation guidance.
Judge whether an ablation pair is a clean test of a claimed variable. Detects unintended config changes (confounds) and returns a verdict (clean/confounded/uncertain) with confidence.
Analyze a hyperparameter sweep to identify which parameters matter most and detect confounding (e.g. correlated hyperparameters). Returns importance rankings and confounding signals.
Analyze a training metric's time-series for anomalies: divergence, crashes, plateaus, and low-variance regions. Returns structured signals and confidence scores.
Diff config and summary metrics across two or more runs to identify what changed.
Fetch the full time-series history of a metric for a single run.
Output schemas not documented. LLMs cannot infer response structure (fields, types, nesting) for audit_training_curve, audit_ablation, audit_sweep. Descriptions mention 'structured signals', 'confidence scores', 'verdict', 'importance rankings' but do not specify JSON shape, field names, or value ranges.
'backend' parameter repeated in 7 of 8 tools but not constrained as enum. Description says 'e.g. wandb' but does not declare valid values. LLMs may hallucinate invalid backends (e.g. 'mlflow', 'tensorboard') causing silent failures.
Descriptions lack actionable context. No guidance on WHEN to call each tool, dependencies between tools, or prerequisites. E.g., audit_ablation does not explain that baseline and ablation must be from same project or that claimed_variable must match a config key.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | <=2025-11-25 | v2 |
Fetch full config and summary metrics for a single run.
List runs in a W&B project with optional filtering by tags, status, and creation date.
Test that W&B credentials are working and the backend is reachable.
list_runs accepts optional filters (tags, status, created_after, created_before, page) but no limit/page_size parameter and no documented pagination behavior. Large projects may return thousands of runs, exhausting context window.
No error handling guidance. Descriptions do not explain what happens if run_id is invalid, metric_name does not exist, or backend is unreachable. LLMs cannot recover from failures without explicit recovery hints.