Loading ToolBench
A lightweight, plugin-oriented LLM pipeline execution kernel for versioned, schema-validated, multi-model analysis pipelines with traceable run bundles.
Server has 8 tools with complete JSON schemas and descriptions. Naming follows verb_noun convention (run_pipeline, get_run_result, list_pipelines). Descriptions are present but brief (avg 80 chars, below 194-char baseline). All tools have typed parameters with descriptions. However, output schemas are not documented in the source code provided, only input schemas are visible. Error handling is mentioned as a feature but no recovery guidance is evident in tool descriptions. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications (WRITE vs READ_ONLY). Parameter descriptions lack format constraints and validation rules.
Export a reproducible run bundle for a stored run.
Fetch the resolved spec and source metadata for a pipeline.
Fetch a stored run result by id.
List available pipeline ids.
List stored run ids.
Re-run a single channel using an existing run's input.
Execute a named pipeline against an input and store the run. To resume a prior partial run, set options.resumeFromRunId (completed channels are carried forward; pipelineId/input are taken from the prior run).
Output schemas not documented. LLMs cannot plan downstream tool calls or extract required fields (e.g., what fields does run_pipeline return? Does it include run_id for chaining to get_run_result?). Baseline: 100% of A+ tools document return types.
Tool annotations missing. Risk classifications (WRITE vs READ_ONLY) are metadata but not exposed as destructiveHint/readOnlyHint in tool definitions. LLMs cannot infer which tools are safe to retry or have side effects.
Descriptions too brief and lack context. 'List available pipeline ids' (40 chars) and 'List stored run ids' (40 chars) do not explain WHEN to call these tools or what structure they reveal. Baseline: avg 194 chars, p10=34, p90=392. These are at p10 floor.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Validate a pipeline definition by id.
Parameter descriptions lack format constraints. 'Input data for the pipeline' does not specify format (JSON string? YAML? Plain text?). 'Timeout in milliseconds' lacks min/max bounds. Baseline: descriptions should state format, range, and allowed values.
No error recovery guidance in descriptions. Tools mention errorReporting=true but descriptions do not explain what errors are possible or how to recover (e.g., 'If validation fails, call validate_pipeline first'). Baseline: error responses must tell LLM what to do next.