Loading ToolBench
Auto-reproduction & certification engine for the computational biomedical modeling literature. Exposes a read-only query surface over the model catalog, certificates, and verification queue via MCP (Model Context Protocol).
Reprolith exposes 18 read-only query tools with consistent naming (verb_noun: list_catalog, get certificate, etc.) and detailed descriptions (avg ~150 chars, well above the 34-char p10 baseline). All tools declare input schemas with required fields and type annotations. However, parameter descriptions are sparse or absent, most params lack guidance on format, constraints, or when to use them. Output schemas are not documented in the visible code, forcing LLMs to infer response structure. Error handling is not visible in the provided source. The domain is highly specialized (biomedical reproducibility), making descriptions dense but sometimes opaque to general-purpose agents. No tool annotations (readOnlyHint, etc.) despite all being read-only.
Backlog depth by state, class, and difficulty, and the labelled mix.
The reconstruction bundle for an entry accession — model, recipe, assumptions.
The full certificate for a digest: content, verdicts, scope, and gaps.
Digests of every certificate issued for a paper, newest first.
What a second, independent simulator said about these verdicts, per model class — and which classes have no second engine, where nothing was checked. Reported beside the certificates, never gating them. An absence is not a pass.
The ingested dossier for an entry accession — its extracted model structure.
Output schemas not documented. LLMs cannot infer response structure (fields, types, nesting). Agents must guess what data is available after each call, risking failed downstream tool chains.
Parameter descriptions missing or minimal. 'digest' param appears in 5 tools with only 'the certificate's content digest', no guidance on format (hex? base64?), length, or how to obtain one. Agents cannot self-correct invalid inputs.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
The structured 'what was missing' report for a digest.
Browse catalog entries as blind public views (no ground-truth label).
Whether there is gated, publishable work — and if not, exactly why not. Assembles the stop condition the autonomous-build-loop spec states: what is claimable, what each blocked entry waits on, what repeated fruitless claims have parked, and what is escalated (split into what an expert can decide and what only this engine can). Reports what stands, not what any one run produced.
Author-facing pre-submission check for a digest: readiness, per-claim verdicts, and a prioritized 'fix before you submit' list. Runs on your own model before publishing.
Which standing certificates owe a re-run, and why. Two causes: an expert corrected a value they rest on (a confirmation owes nothing — the numbers do not move), and the judging code has moved since they were issued, so their numbers are not the ones this checkout produces. Dependents of a rejected estimate are reported as blocked rather than due: a rejection supplies no replacement value. It re-runs nothing.
Which of a paper's claims to reproduce within a budget, when the budget will not cover them all. Maximizes independent evidential value — item value less the overlap between claims resting on the same parameters, model components, and upstream assumptions — and reports the plain one-at-a-time ranking beside it. A plan, not a verdict: no model is run and no certificate is issued.
Reprolith's blind self-validation track record — per model class and overall, how its verdicts matched independently-established ground truth. The evidence a verdict rests on.
A paper's lifecycle status and recorded history, by identifier.
The scope-qualified verdict for a digest (never a bare boolean).
One verification-queue item as the GitHub issue to open: title, the three labels the collaboration spec asks for (model class, impact rank, pending-verification status), and a body carrying the question, the estimate and its basis, the alternatives, and the papers that rest on it. Refuses an item that is a limit of this engine rather than a question a paper could answer — no expert decision closes one. It files nothing; it returns the issue for a human to open.
The load-bearing values every standing certificate rests on, most consequential first: the question, Reprolith's best estimate, its basis, the alternatives, and which certificates would have to be re-issued if an expert corrected it.
This server's package version and the judge revision each model class publishes under — the digest over the code that computes a verdict, which every certificate names and which expires when that code changes. An agent recording a lint verdict records what produced it with this.
No tool annotations despite all 18 tools being read-only. Missing readOnlyHint annotation prevents agents from optimizing caching and retry strategies. Agents cannot distinguish safe-to-retry tools from stateful ones.
Identifier parameters (title, doi, pubmed_id, accession) lack constraints or format guidance. Agents cannot validate inputs before calling. No guidance on which identifier to prefer or how to resolve ambiguities (e.g., 'AmbiguousMerge' exception in catalog).
Error handling not visible in source. No guidance on recovery (e.g., what to do if digest is invalid, accession not found, or budget exceeds available claims). Agents cannot self-correct or escalate intelligently.