Loading ToolBench
vmbench is a formal VM benchmark and synthetic training/eval toolkit. Use it to generate benchmark datasets, run bounded baseline evaluation, evaluate gate summaries, and export SFT prompt/completion data.
17 tools with mostly complete schemas and descriptions, but significant gaps in parameter documentation and error handling guidance. Tool names follow verb_noun convention well (vmbench_*). Descriptions average ~120 chars, below the 194-char baseline for A+ tools. Most parameters have type definitions and descriptions, but many lack constraints (enums, ranges, format specs). No output schemas documented. Error handling is minimal, tools do not guide recovery or categorize errors. Security considerations present (READ_ONLY/WRITE risk labels) but not formalized in descriptions. Composition is reasonable, tools are single-purpose, but chaining requirements are not explicit.
Choose the next instruction step given a program state and candidate instructions.
Compare multiple ranking policies on a benchmark record.
Compare two baseline summary reports.
Generate demo reasoning runtime payload for interactive visualization.
Analyze and explain why a benchmark record failed to solve.
Export SFT (Supervised Fine-Tuning) prompt/completion pairs from dataset.
Evaluate a baseline summary against curriculum gates.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields without knowing response structure.
Parameters lack constraint documentation (enums, ranges, formats). E.g., 'ranker_strategy' accepts 'none'|'heuristic'|'learned' but no enum constraint; 'stages' array items lack format spec; numeric limits (node_budget, candidate_limit) have no min/max bounds.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Generate benchmark dataset with single_step, next_2_steps, short_trace, and terminal_state examples.
Generate demo payload for a single benchmark record.
Retrieve the log of all LLM tool calls made during this session.
Load and return the vmbench manifest (dataset metadata, task library, splits).
Load and return the repository product map.
Run baseline evaluation on a dataset using Ollama model.
Evaluate a checkpoint model on a dataset.
Set reasoning text to be attached to the next tool call.
Solve a benchmark record with a given node exploration budget.
Get current server status and configuration.
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. E.g., vmbench_run_baseline may fail if Ollama is unavailable, but no recovery hint is provided.
vmbench_status and vmbench_set_next_reasoning have minimal descriptions (<60 chars). Insufficient context for LLM selection. vmbench_status lacks explanation of what 'status' means; vmbench_set_next_reasoning lacks guidance on when/why to use it.
Tool chaining requirements not documented. E.g., vmbench_choose_next_step requires 'program_name' and 'before_state_text', where do these come from? Does vmbench_solve_with_budget produce them? No cross-tool guidance.