Background computer control via focus-free UI automation. Controls apps without moving the real cursor, changing focus, or stealing keyboard. Uses UI Automation patterns or window messages to interact with elements on occluded/background windows.
Desktop Computer Use (DCU) provides 7 tools for focus-free desktop automation. Tool names are clear action-verbs (list_apps, get_app_state, click, type_text, scroll, set_value, hide_cursor). Descriptions are present and substantive (average ~180 chars), explaining WHAT each tool does and WHEN to use it. Input schemas are well-defined with types, descriptions, and sensible defaults. However, there are gaps: (1) Output schemas are not formally documented in code, responses are constructed ad-hoc with new{} objects, making it impossible for LLMs to know what fields to expect until they see a response; (2) Error handling is minimal, most validation delegates to ToolValidation.ValidateMaxElements/ValidateClick, but recovery guidance is sparse; (3) Security patterns are underspecified, no audit logging, permission gates, or scope declarations visible in the tool definitions; (4) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels (click/type_text/scroll/set_value are WRITE, while list_apps/get_app_state are READ_ONLY). These gaps prevent confident production deployment but do not disqualify the server, the core definitions are solid.
Focus-free click. Prefer element_id from the latest snapshot; or use screen x,y. Uses UIA Invoke/Toggle/Select patterns when available (left clicks), else posts window messages. Never moves the real cursor or changes focus.
Snapshot an app's accessibility tree: interactive elements with stable ids, their frames, and supported actions. Optionally includes an annotated screenshot (Set-of-Marks labels matching element ids). Works on background/occluded windows.
Hide the virtual cursor overlay (the green ghost cursor shown during actions).
List running apps that have a visible main window. Returns name, pid, title.
Focus-free scroll. Prefers UIA ScrollPattern; falls back to key events (Chromium) or PostMessage wheel events. Scrolls by pages or pixels.
Focus-free value assignment. For edit controls: EM_SETSEL + EM_REPLACESEL. For UIA ValuePattern: SetValue (with optional fallback). Never focuses.
Output schemas are not formally documented. Tool responses are constructed via ad-hoc new{} objects with no schema declarations. LLMs cannot know what fields to expect from list_apps, get_app_state, click, etc. until they receive a live response. This violates pattern:tool and pattern:response-shaper, tools must declare their output structure so agents can plan downstream calls.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels declared in the metadata (READ_ONLY vs WRITE). Spec 2026-07-28 includes tool annotation extensions for destructive/readonly/idempotent hints. These should be wired into the McpServerTool attributes or a separate descriptor to help agents reason about safety.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Focus-free text entry. Cascades through EM_REPLACESEL (edit controls), key-event streams (Chromium/Electron), UIA ValuePattern.SetValue (with fallback option), and WM_CHAR messages. Handles unicode, RTL, and long text.
Error handling is minimal and non-actionable. Validation errors (e.g., 'Invalid status: got "pending", must be...') are generated by ToolValidation helpers, but there is no visibility into what those helpers return. The code shows Error() being returned but no documented error taxonomy, retry guidance, or user-fixable vs fatal classifications per pattern:error-classification.
Security patterns underspecified. No audit logging visible (who called this tool, when, with what parameters, and what happened). No permission gates or scope declarations. Tools like click and type_text modify system state, they should require explicit authorization and log actions for compliance. Pattern:audit-trail and pattern:permission-gate are not evident.
hide_cursor description is too brief (60 chars: 'Hide the virtual cursor overlay...'). This tool's purpose is cosmetic, when would an agent call this instead of leaving the cursor visible? Expand description to explain the use case.
Parameter descriptions lack format/range constraints. E.g., 'max_elements' has no explicit min/max bounds stated in the description (though ToolValidation.ValidateMaxElements suggests validation exists).