A macOS computer use agent (CUA) that automates tasks through screen analysis, action execution, and multi-step planning with LLM backends
TuriX-CUA is a macOS automation agent with 14 tools spanning UI automation (Click, Drag, scroll_up/down), system control (open_app, run_apple_script, Hotkey), and task management (record_info, wait, done). Tools are defined in src/agent/output_schemas.py with basic JSON Schema. However, CRITICAL ISSUES severely limit quality: (1) Tool names violate verb_noun convention, 'RightSingle', 'move_mouse', 'scroll_up/down', 'multi_Hotkey' are inconsistent and lack clarity. 'RightSingle' is ambiguous (does it right-click once? toggle?). 'scroll_up' and 'scroll_down' are redundant when a single 'scroll' tool with signed 'dy' parameter would suffice. (2) Descriptions are minimal (10-30 chars) and lack WHEN-to-use guidance. E.g., 'Click at a specified position on screen' does not explain when to use Click vs RightSingle, or what happens if coordinates are invalid. (3) Input schemas present but incomplete: no type constraints on 'position' array (should specify array length=2, items=number in range [0,1000]). The 'dy' and 'dx' parameters in scroll tools lack min/max bounds. (4) No output schemas documented anywhere. Tools return success/failure implicitly but LLM has no structure to parse. (5) No error handling guidance, what if a click misses a UI element? What if the app fails to open? (6) No composition aids, tools like scroll_up/down and move_mouse operate at primitive levels; no higher-level tools like 'find_element_and_click' or 'navigate_to_url'. (7) Security: run_apple_script accepts arbitrary script strings with no sanitization hints; record_info writes files with user-controlled paths (path traversal risk). Code review in examples/main.py shows LangChain integration but no MCP server scaffolding visible, transport and protocol details unknown.
Click at a specified position on screen
Drag from one position to another on screen
Press a single hotkey combination
Right-click at a specified position on screen
Mark the task as completed
Type text at the current cursor position
Move mouse cursor to a specified position
Press multiple hotkey combinations in sequence
Open a macOS application by name
Tool names violate verb_noun convention and contain ambiguous terms. 'RightSingle' is cryptic (means right-click once?). 'multi_Hotkey' uses underscore inconsistently. 'scroll_up'/'scroll_down' are semantically redundant (use single 'scroll' with signed dy). Names do not reliably convey action intent to LLM.
Descriptions are too brief (<35 chars average) and lack WHEN-to-use guidance. 'Click at a specified position on screen' does not explain when to prefer Click over RightSingle, or how LLM should interpret normalized coordinates. Descriptions must answer WHAT, WHEN, PREREQUISITES.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 43 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 27 | - | v1 |
Record information to memory/file for later retrieval
Execute an AppleScript command
Scroll down at a specified position
Scroll up at a specified position
Wait for a specified condition or time
Input schemas present but missing constraints. 'position' parameter is [number, number] but lacks itemType, length validation, and range [0,1000]. 'dx'/'dy' in scroll tools have no min/max bounds, inviting invalid values. 'script' in run_apple_script has no length limit or sanitization hints.
No output schemas documented. Tools return implicit success/failure but LLM has no structured format to parse results, chain to downstream tools, or extract IDs for follow-up operations.
No error handling guidance. If click misses target, app fails to open, or script execution times out, LLM receives no actionable recovery guidance. Missing recovery_guide pattern.
Security gap: run_apple_script accepts arbitrary script strings with no sanitization hints or injection warnings. record_info writes to user-controlled file paths without path traversal protection documentation.
Primitive tool composition. Tools like scroll_up, move_mouse, and Click operate at low automation level. No higher-level tools like 'find_element_and_click', 'navigate_to_url', or 'fill_form' exist to reduce multi-step chains and improve efficiency.
Redundant scroll tools. scroll_up and scroll_down duplicate logic, a single 'scroll' tool with signed 'dy' parameter (positive=down, negative=up) and optional 'dx' would be clearer and reduce naming confusion.