Loading ToolBench
An MCP server for managing document corpus ingestion, retrieval pipeline experiments, and evaluation of retrieval systems
Three tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. Tool names follow verb_noun convention (run_experiment, get_run_summary, get_runs), which is good. However, descriptions are minimal (10-50 chars), parameter descriptions lack detail about constraints and formats, and output schemas are not documented. No error handling guidance, no parameter validation rules, and no indication of idempotency or side effects. Schemas are present but sparse, parameters have types but lack constraints (enums, ranges, patterns). This is typical of a D-grade server: functional but not production-ready.
Get the summary of a completed or in-progress experiment run
List all experiment runs, optionally filtered by spec_id or status
Run an experiment based on a specification file
Descriptions are too short (10-50 chars) and lack context. 'Run an experiment based on a specification file' does not explain WHEN to use this tool, what prerequisites exist, or what the return value contains. LLMs cannot reliably select this tool without richer guidance.
Output schemas are not documented. Callers cannot know what fields run_experiment returns, what structure get_run_summary produces, or how to chain results into downstream tools. This forces LLMs to guess and wastes tokens on discovery.
Parameter descriptions are missing or generic. 'spec_path' has no guidance on format (file path? URL? relative or absolute?). 'artifacts_dir' defaults to 'artifacts' but no description explains what this directory contains or whether it must exist. 'status' in get_runs lists valid values in the description but should use an enum constraint instead.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
No error handling guidance. If run_experiment fails (invalid spec, missing file, permission denied), the tool provides no recovery hints. LLMs cannot self-correct or know whether to retry, ask the user, or abandon the task.
No indication of idempotency or side effects. run_experiment clearly modifies state (writes artifacts), but the description does not say so. Agents need to know which calls are safe to retry and which have irreversible consequences.