Loading ToolBench
macOS UI automation and computer use server exposing tools for GUI interaction, window state inspection, recording, and accessibility control
This server demonstrates strong naming conventions and comprehensive schema definitions across all 10 tools. Tool names follow verb_noun patterns (check_permissions, get_accessibility_tree, get_cursor_position). All tools have input schemas with type definitions and descriptions. However, descriptions vary significantly in quality and completeness, some tools have excellent docstrings (click, drag, double_click) while others are minimal (get_cursor_position, get_screen_size). Output schemas are not explicitly documented in the code sample, which is a gap. Error handling is present but recovery guidance is limited. The server demonstrates good parameter constraint usage (enums for actions, min/max bounds on numeric parameters like duration_ms 0-10000, steps 1-200, count 1-3). Accessibility to native system primitives (AX framework, CGEvent) is well-explained in tool descriptions. No security concerns detected, no credentials as parameters, permission gating present (check_permissions tool). Tool composition is clean: each tool has a single responsibility. However, the rubric expects documented output schemas for LLM planning, which are absent.
Report TCC permission status for Accessibility and Screen Recording. By default also raises the system permission dialogs for any missing grants — Apple's request APIs are no-ops when the grant is already active, so this is safe to call repeatedly. Pass {"prompt": false} for a purely read-only status check.
Left-click against a target pid. Two addressing modes: - `element_index` + `window_id` (from the last `get_window_state` snapshot of that window) — performs an AX action on the cached element. Default `action` is `press` (left-click). Other values: `show_menu` (right-click equivalent), `pick` (open a menu-bar item's submenu), `confirm` (return on a default button), `cancel` (escape on a dismiss button), `open` (open a file/folder in a browser). Pure AX RPC, works on backgrounded / hidden windows, no cursor move or focus steal. Requires a prior `get_window_state(pid, window_id)` in this turn; the element_index cache is scoped per (pid, window_id) so indices from one window do not resolve against another. - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG returned by `get_window_state`) — synthesizes mouse events via CGEvent / SkyLight and delivers them to the pid. AX is never consulted on this path. The driver converts image-pixel → screen-point internally using the target's window origin and backing scale. `count: 2` posts two down/up pairs ~80ms apart for a double-click. `modifier` holds cmd/shift/option/ctrl during the click (e.g. ["cmd"] for cmd-click). Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` is required in both modes. `window_id` is required when `element_index` is used (scopes the cache lookup). `action` is only valid with `element_index`; `count` and `modifier` are ignored in the AX path.
Double-click against a target pid. Two addressing modes: - `element_index` + `window_id` (from the last `get_window_state` snapshot of that window) — performs `AXOpen` when the element advertises it (Finder items, list rows, openable cells); otherwise falls back to a stamped pixel double-click at the element's on-screen center. Pure AX RPC in the AXOpen branch; pixel recipe when no AX open action exists. Requires a prior `get_window_state(pid, window_id)` in this turn. - `x`, `y` (window-local screenshot pixels, top-left origin of the PNG returned by `get_window_state`) — synthesizes a stamped double-click (two down/up pairs with clickState 1→2) and delivers to the pid. The primer-gated auth-signed recipe lands on backgrounded Chromium web content (dblclick events that trigger YouTube fullscreen toggle, Finder open-on-double-click, etc.). `modifier` holds cmd/shift/option/ctrl during the gesture. Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` is required in both modes. `window_id` is required when `element_index` is used. `modifier` only takes effect in the pixel path (AXOpen doesn't propagate modifier keys).
Output schemas not documented. LLMs cannot plan downstream actions without knowing what fields to expect. All 10 tools lack explicit output schema documentation, only input schemas are visible in code.
Minimal descriptions on discovery/read tools. get_cursor_position ('Return the current mouse cursor position in screen points...') is only 72 characters, under the 10-1024 character baseline and lacks context on when/why an LLM should call it. get_screen_size similarly minimal at 88 chars. These violate the pattern:tool-description guideline that descriptions must explain WHAT, WHEN, and WHY.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Press-drag-release gesture from (`from_x`, `from_y`) to (`to_x`, `to_y`) in window-local screenshot pixels — the same space the PNG `get_window_state` returns. Top-left origin of the target's window. Use this for: marquee/lasso selection, drag-and-drop between source and destination, resizing via a handle, scrubbing a slider, repositioning a panel. macOS AX has no semantic drag action, so drag is pixel-only — there is no element-indexed mode. Address an element with `get_window_state`, read its `bounds`, and pass the pixel coordinates you want. `duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate `mouseDragged` events linearly interpolated along the path. Increase both for slower, more "human" drags; decrease for snap gestures. `modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture — option-drag duplicates, shift-drag constrains the axis on most surfaces. Frontmost target: posts via `.cghidEventTap`. The real cursor visibly traces the drag path — unavoidable for AppKit-style drag sources, which accept only HID-origin events. Backgrounded target: posts via the auth-signed pid-routed path. Cursor-neutral, but some surfaces (OpenGL canvases, Blender, Unity) filter pid-routed dragged events at the event-source level — those targets must be frontmost. When `from_zoom` is true, both endpoints are pixel coordinates in the last `zoom` image for this pid; the driver maps them back to window coordinates before dispatching.
Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. For the full AX subtree of a single window (with interactive element indices you can click by), use `get_window_state` instead — that's the heavy per-window tool. This one is a fast discovery read that needs no TCC grants.
Report the current agent-cursor configuration: enabled flag, motion knobs (startHandle, endHandle, arcSize, arcFlow, spring), glide duration, post-click dwell, and idle-hide delay. Durations come back in milliseconds to match the setter's units. Pure read-only — no side effects.
Report the current persistent driver config. Config lives at `~/Library/Application Support/<app-name>/config.json` and survives daemon restarts, unlike session-scoped state (recording). Pure read-only. Returns defaults when the file doesn't exist or fails to decode — same fallback the daemon uses at startup. Sibling to `set_config` / `trope-cua config`. Current schema: { "schema_version": 1, "capture_mode": "vision" | "ax" | "som", "agent_cursor": { "enabled": true, "motion": { "start_handle": 0.3, "end_handle": 0.3, "arc_size": 0.25, "arc_flow": 0.0, "spring": 0.72 } } }
Return the current mouse cursor position in screen points (origin top-left).
Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled. Pure read-only. Typical use: `trope-cua recording status` surfaces these fields for human consumption; programmatic callers use it to confirm recording is armed before kicking off a session.
Return the logical size of the main display in points plus its backing scale factor. Agents click in points; Retina displays have scale_factor 2.0. Requires no TCC permissions.
Inconsistent description depth across similar tools. get_accessibility_tree (125 chars) vs get_cursor_position (72 chars) vs get_screen_size (88 chars). While all are functional, the variance suggests inconsistent documentation standards. Tools operating at similar abstraction levels should have comparable description quality.
Error recovery guidance absent. While the code shows error paths (e.g., MCPError.invalidParams), the tool descriptions do not guide LLMs on recovery. E.g., click tool description does not explain what to do if the pid is invalid or the element_index is stale. Pattern:recovery-guide requires explicit 'Try X instead' hints.
Permission requirements not declared in tool descriptions. check_permissions mentions TCC (Transparency, Consent & Control) but other tools (click, drag, get_accessibility_tree) do not declare which permissions they need. Pattern:scope-declaration requires each tool to declare its permission scope.