Loading ToolBench
Model-agnostic computer-use core exposed as a Model Context Protocol server. Provides native tool-calls for GUI automation with normalized coordinates, canonical action schema, and a central safety gate. Supports Claude, OpenAI CUA, and mock backends with process persistence, native image return, and safety confirmation workflow.
13 tools with complete schemas and descriptions, but significant gaps in parameter documentation and error handling guidance. Tool names are action-oriented (capture, do, click_name, invoke, list_windows, describe_window, rec_replay, signal_*, observation_*), following verb_noun convention well. Descriptions range 50 - 250 chars, meeting baseline (194 avg). However, many parameters lack type constraints (enums, ranges), and error responses do not guide recovery. Safety-critical tools (do, click_name, invoke, signal_abort) lack explicit confirmation/dry-run patterns despite irreversible consequences. Output schemas are undocumented, LLMs cannot predict response structure for chaining.
Capture the current screen or a specific window region. Returns a screenshot as an MCP image block (PNG bytes). Coordinates are normalized 0..1. Window-local captures require the physical source frame and window identity.
Click on a UI element by semantic name (UIA automation element or set-of-marks label). Resolves the element within an optional window context and executes a click. Runs through the safety gate.
Get detailed information about a specific window: title, class, process, UIA element tree, and semantic observations. Useful for understanding window structure before interaction.
Execute a canonical action (click, type, scroll, etc.) on the desktop. Actions use normalized coordinates 0..1. Supports mouse actions (click, drag, move, scroll), keyboard (type, key press), window management (launch_app, activate_window), and hold primitives (mouse_down/up, key_down/up). Runs through the safety gate; may return needs_confirmation if safety mode is 'confirm'.
Invoke a UIA automation element by name with a specific pattern (e.g., Invoke, Toggle, Expand). Resolves the element and calls its automation pattern method. Runs through the safety gate.
Output schemas undocumented. LLMs cannot predict response structure (e.g., what fields does capture return? what does do return on needs_confirmation?). Breaks tool chaining and forces agents to guess field names.
Parameter constraints missing. 'action' in do tool accepts free-form string instead of enum (screenshot, mouse_move, left_click, etc.). 'visual_mode' in capture accepts free-form instead of enum (raw, marked, semantic, marked+semantic). Invites hallucinated invalid values.
Error handling lacks recovery guidance. No documented error responses or categorization (retryable vs fatal). Tools return 'needs_confirmation' but no schema for that response. Agents cannot distinguish transient failures from permission denials.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
List all top-level windows currently open on the desktop. Returns window titles, handles, foreground state, and stable window tokens for use in subsequent operations.
Append a note or observation to the model-to-human observation overlay. Used in work-together mode to communicate findings to the human operator.
Hide the model-to-human observation overlay.
Show the model-to-human observation overlay (notes window). Displays a persistent diary of observations and notes for the duration of the session.
Record or replay a sequence of actions. In record mode, captures a baseline and monitors for file changes via dirwatch. In replay mode, executes a previously recorded action sequence. Runs through the safety gate.
Trigger the emergency stop (kill switch). Latches the abort state so all gate-relevant tools deny outright until a fresh signal_show re-arms the session.
Hide the human-in-the-loop signal overlay. Cancels any pending grace period or lease timer.
Show the human-in-the-loop signal overlay (action confirmation border). Displays a visual indicator on screen and arms a grace period before actions can execute. Used for safety confirmation workflow.
Irreversible operations (signal_abort, do with destructive actions) lack explicit confirmation or dry-run pattern. signal_abort 'latches abort state', no undo mechanism documented. Agents cannot safely preview consequences.
Numeric parameters unbounded. 'scroll_amount' in do has no min/max. 'lease_seconds' in signal_show has no range. 'duration' in do (wait action) has no upper bound. Agents can pass absurd values (wait 999999 seconds).