Loading ToolBench
Bioinformatics harness. Use probe_data / list_catalog / explain_tool / find_tool to gather facts and pick a tool. To RUN anything, call run_tool — it self-guards (onboarding + judgment) and may refuse. Never assume a run is allowed; read run_tool's returned trace.
bio-harness defines 9 tools with clear, domain-specific purposes and generally good descriptions (avg 156 chars). All tools have input schemas with typed parameters and descriptions. However, output schemas are not documented in the visible code, and several tools lack parameter constraints (enums, ranges). Tool naming follows verb_noun convention well (probe_data, list_catalog, run_tool). The server is well-architected for bioinformatics workflows but falls short of production-grade polish in schema completeness and error guidance.
Diagnose a hard failure: match the crash's stderr/stdout/error against the tool's known failure_modes; escalate novel crashes.
Score a completed run's output directory against the tool's expectation table. Refuses with 'cannot_assess' if output can't be parsed.
Curated facts about one documented tool (summary, usage, options, off-label boundaries, citation). Deterministic; the client narrates.
Find candidate tools for a natural-language need (deterministic catalog search).
Judge a spec against a tool's contract (HRR gate + preconditions + boundaries). Returns a route with the right to REFUSE. For transparency; run_tool judges internally regardless.
List/filter the documented tool catalog. All args optional.
Output schemas not documented. Tools return results but LLMs cannot predict response structure, forcing trial-and-error parsing and wasting tokens.
Parameter constraints missing. 'provider' param appears in 5 tools but has no enum of valid values (claude, ollama, etc.). 'category' and 'input_format' in list_catalog lack enums. LLMs will hallucinate invalid values.
Error handling guidance absent. Tools like 'judge' and 'diagnose_failure' can refuse or escalate, but descriptions do not explain what the LLM should do next (retry, ask user, abort).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Build the experiment spec for a file: probe measured facts, extract declared facts from the question, reconcile, and return disagreements. (Single-file; multi-file experiment documents are a later phase.)
Measured facts about a data path (FASTQ file, alignment, or report dir). Read-only; no interpretation.
THE execution entrypoint. Self-guarding: runs onboarding -> judgment -> (refuse | execute -> evaluate | diagnose) and returns the full trace. A refusal means nothing was executed. This is the only way to run a tool.
Parameter 'spec' in judge tool is typed as 'object' with no schema. LLMs cannot construct valid experiment specs without seeing the expected structure.
No pagination or result limits documented. 'list_catalog' and 'find_tool' may return large result sets; no mention of limit, offset, or total_count fields.