Loading ToolBench
MCP server for macOS computer automation via UI interactions. Provides tools to interact with desktop apps by performing clicks, scrolling, typing, and other UI actions.
Strong tool definitions with comprehensive descriptions and well-structured schemas. All 12 tools have clear verb-noun naming (list_apps, get_app_state, click, etc.) and detailed descriptions (avg 180+ chars) explaining WHAT, WHEN, and HOW. Input schemas are present with typed parameters and descriptions. However, output schemas are not documented, LLMs cannot predict return structure. Error handling is present but lacks recovery guidance. Some parameter descriptions could be more prescriptive about constraints (e.g., coordinate_space enum values are documented but coordinate ranges are not). Tool composition is excellent: tools chain well (click returns updated observation), and stable_ref/element_text/element_index targeting is sophisticated. No security issues detected (no secrets in params). Descriptions are LLM-optimized and avoid example values.
Click an element. Prefer stable_ref from the latest daemon/session-backed snapshot/diff when available, and pair it with element_text when possible so a replaced AX node can safely fall back to text matching. Otherwise provide BOTH element_text and element_index for robust targeting — element_text is the stable anchor, element_index is the fast path verified against it. If element_index is stale (doesn't match element_text), the tool automatically falls back to text search. Every action tool returns the updated observation + screenshot with a snapshot ID: AXDIFF v1 when daemon/session-backed and safe, otherwise a full AX tree, so you do NOT need a separate get_app_state call after. Use click_count=2 for double-click, 3 for triple-click. Omit app to click at screen-level coordinates from get_screen_state (requires x and y). Coordinate inputs default to screenshot pixels; set coordinate_space when your model emits normalized coords. For screen-level x/y (app omitted), the tool tries to AX-snap to an actionable element under that point before raw input fallback. App-level x/y uses coordinate input fallback directly. WARNING: Context menus (right-click menus) are transient — using element_text or app-level click triggers an AX tree walk that dismisses the menu. To click a context menu item, use screen-level coordinates (x, y without app) from the screenshot instead.
Double-click an element or screen location. Equivalent to click with click_count=2.
Drag from one point to another. Coordinates default to screenshot pixels; set coordinate_space when your model emits normalized coords. Omit app to drag at screen-level coordinates from get_screen_state. Returns updated state.
Output schemas not documented. LLMs cannot predict return structure (AX tree format, AXDIFF v1 structure, screenshot format, field names). Forces agents to guess or fail on parsing.
Coordinate constraints not fully specified. Parameters x, y, from_x, from_y, to_x, to_y lack min/max bounds or pixel range documentation. LLMs may pass out-of-bounds values.
Error recovery guidance missing. Descriptions explain what tools do but not what to do if they fail (e.g., 'If element not found, try get_app_state first' or 'Retry with stable_ref instead of element_index').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2025-06-18+ | v2 |
Snapshot the app's key window: full AX tree with ephemeral element indices, daemon/session-backed stable refs when available, and screenshot. Launches app if not running. NOTE: You often do NOT need this — action tools (click, scroll, type_text, etc.) return the updated observation, usually as AXDIFF v1 when daemon/session-backed and safe. Only call this when you need to inspect the UI without performing an action. WARNING: Do NOT call this while a context menu (right-click menu) is open — the AX tree walk will dismiss the menu. Use the screenshot from the action that opened the menu and click by screen coordinates instead.
Capture a full-screen screenshot of the main display with visible window positions and hidden/minimized apps. Use this to see the desktop layout, system UI (menubar, Dock), or find which apps are visible. For inspecting a specific app's UI elements, use get_app_state instead. Coordinates from this screenshot can be used with click and drag (without app parameter) for screen-level interactions.
Move the pointer over an app element or app-screenshot coordinate and keep it there. Use when a control reveals menus, tooltips, or child content on hover. Returns updated observation.
List running and recently used apps.
Invoke a secondary AX action (e.g. AXShowMenu for context menu, AXRaise), or action=activate_app to bring the target app frontmost and raise its window. Returns updated observation.
Press a keyboard key or key combination. Returns updated observation.
Scroll within an app or at screen level. Returns updated observation.
Set the value of a control element. Returns updated observation.
Type text into the focused element or app. Returns updated observation.
double_click is redundant with click(click_count=2). Separate tool adds cognitive load without new capability. Should be removed or documented as a convenience alias.
perform_secondary_action description lacks examples of valid action names (AXShowMenu, AXRaise, activate_app). LLMs cannot infer valid values without enum or examples.