Loading ToolBench
Test that a bot only ever does the job it was given — and never deletes, sends or approves the thing it must not. Record it doing the job once, then replay on every commit with zero LLM calls. For the agents and RPA already running real work in production: web, desktop, SAP, Citrix.
Four tools with detailed descriptions (194 - 350 chars, above baseline 194) and comprehensive input schemas. All parameters have types and descriptions. However, output schemas are documented only in descriptions, not as formal JSON Schema. Error handling guidance is present but inconsistent, flowproof_record and flowproof_heal describe recovery paths (needs_clarification, proposed.jsonl), but flowproof_run and flowproof_get_trace lack explicit error recovery guidance. Tool naming is clear and verb-forward (record, run, get, heal), but composition could be tighter: flowproof_record and flowproof_heal both author flows, creating potential confusion about when to use each. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles (WRITE vs READ_ONLY).
Load a recorded trace for inspection. `path` may be the flow spec (the default trace next to it is used) or a .jsonl trace file. Returns {"header", "steps"}.
Re-author the flow against the live app and propose a reviewable trace diff (written as *.proposed.jsonl). Never modifies the trace unless apply=true is passed explicitly. Returns {"report": {"changed", "steps_changed", ...}, "applied"}; author accepts auto, rules, or llm.
Record a flow: perform it once against the live app and write a deterministic trace. Requires the target platform (Windows for UIA apps like calc/notepad; any OS for `app: web`). `author` accepts auto, rules, or llm. Returns {"trace_path", "steps", "reused_steps", "routing"} on success, OR {"needs_clarification": {step, step_index, stage, reason, capture_reference?, capture_candidates?, rules_error?, completed_steps, scene: [{target, tag?, label?, text?, type?}], hint}} when a step is too ambiguous to author — the scene is the live screen's actionable/readable inventory: rewrite the stuck step into concrete grammar (docs/authoring.md) targeting a listed element, consulting your data source for domain questions (e.g. which fields are required), then call this tool again. A spec whose skip_unless_env gate is not satisfied returns {"skipped": reason} — nothing was recorded. recording_detail (full/low/off), video and highlight_cursor are the visual-capture controls described on flowproof_run.
Deterministically replay a recorded flow (zero LLM calls). A failing test is data ({"report": {"passed": false, ...}}), not an error. Returns {"report", "report_path"}; a spec whose skip_unless_env gate is not satisfied returns a skipped report with {"report_path": null, "skipped": reason} — the flow never ran. recording_detail picks the visual-capture density: "full" (before and after every step), "low" (initial state, every fifth completed step, final state), or "off". video assembles the checkpoints into recording.gif (off by default); highlight_cursor draws a visible cursor and click halo. All three affect artifacts only, never the verdict.
Output schemas documented in descriptions only; no formal JSON Schema return types visible. LLMs cannot parse expected response structure programmatically.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint) despite explicit risk labels (WRITE, READ_ONLY). Agents cannot infer safety properties from schema alone.
flowproof_record and flowproof_heal both author flows (author param in both). Unclear when to use each, record is initial authoring, heal is re-authoring, but descriptions do not explicitly state this distinction.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
flowproof_run and flowproof_get_trace lack explicit error recovery guidance. flowproof_record and flowproof_heal describe recovery paths (needs_clarification, proposed.jsonl), but run/get_trace do not.
Parameter 'author' accepts 'auto, rules, or llm' but no enum constraint visible in schema. Free-form string invites hallucinated values.