MCP server providing accessibility tree manipulation, UI automation, memory management, and procedure replay tools for an AI agent running on macOS. Includes AX-based tools for reading/controlling the accessibility tree, visual inspection, timeline search, and persistent directive management.
Shadow Agent Tools demonstrates strong naming conventions (verb_noun pattern across all 26 tools) and comprehensive parameter descriptions. All tools have clear, actionable descriptions (avg ~150 chars, within 10-1024 baseline). Input schemas are well-structured with type definitions and parameter constraints (enums, min/max bounds). However, output schemas are not explicitly documented in the source code, responses are inferred from handler implementations. Error handling is present but lacks recovery guidance (e.g., 'Try search_users() first'). Security is well-designed with no exposed credentials in parameters. Tool composition is excellent: single-responsibility design, proper chaining via ID returns (e.g., procedureId, bundleId). Minor gaps: some tools lack explicit enum constraints (e.g., ax_scroll direction could be stricter), and output field naming consistency could be formalized.
Click a UI element. Finds the element by query text, role, or screen coordinates. Uses AX-native click first (no focus steal), falls back to synthetic mouse click. Supports double-click and right-click.
Find the UI element at specific screen coordinates. Returns the element's role, title, and metadata.
Bring an application to the foreground by bundle ID or name. Activates the app and makes it the frontmost window.
Press a keyboard shortcut. Supports modifier keys (cmd, shift, alt, ctrl) combined with regular keys.
Get detailed metadata about a UI element: role, title, value, identifier, description, frame, supported actions, and parent/child relationships.
List all running applications with their bundle IDs, names, and process IDs. Useful for finding apps to focus or interact with.
Output schemas not explicitly documented in source code. Responses are inferred from handler implementations (formatJSONLine, AXSearchResult structures). LLMs cannot reliably predict response fields without formal schema documentation.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | <=2025-11-25 | v2 |
Read text content from the focused element or a specific element by query. Returns the full text value or visible text content.
Scroll a scrollable element or the focused window. Supports directional scrolling and magnitude control.
Read the accessibility tree of the frontmost application. Returns interactive UI elements (buttons, text fields, links, etc.) with their roles, titles, positions, and available actions. Use this to understand what's on screen before clicking or typing.
Type text into the focused element. Supports special keys (Return, Tab, Escape, etc.) via key names in curly braces.
Wait for a UI element to appear or disappear. Polls the accessibility tree until the condition is met or timeout expires.
Capture a screenshot of the current screen or a specific display. Returns base64-encoded image data.
Get a sequence of user activities within a time range. Returns ordered list of app switches, window changes, and interactions.
Get a summary of the user's activity for a specific day. Returns high-level overview of apps used, time spent, and key events.
List active directives (reminders, automation triggers, watches). Returns all non-expired, active directives sorted by priority. Use this to check what persistent instructions are set up.
Query the user's long-term semantic memory. Returns learned facts, preferences, patterns, and behavioral knowledge. Use this to personalize responses and understand the user's habits. Categories: preference, fact, pattern, relationship, skill.
List available automation procedures. Returns procedure names, descriptions, and metadata for procedures that can be replayed.
Get timeline context for a specific timestamp. Returns surrounding events, app activity, and window information.
Get transcript text from a specific time window. Returns audio transcription and speaker information for the requested time range.
Inspect screenshots using OCR and optional vision language model analysis. Returns text content and visual descriptions.
Replay a saved automation procedure. Executes a sequence of recorded UI actions with optional parameter substitution.
Resolve the latest meeting from calendar or transcript data. Returns meeting details including participants, time, and content.
Search the user's captured timeline for apps, windows, OCR text, and transcripts. Returns matches ranked by relevance. Supports optional time range filtering.
Search summaries of past activities and events. Returns relevant summaries ranked by relevance to the query.
Search visual memories using CLIP embeddings. Returns screenshots and frames ranked by semantic similarity to the query.
Create a directive — a persistent instruction for the agent. Types: 'reminder' (trigger-based reminders), 'habit' (recurring behavioral nudges), 'automation' (auto-run a procedure when condition matches), 'watch' (monitor and alert). Directives persist across sessions and can have optional TTL.
Error handling lacks recovery guidance. Errors return generic messages (e.g., 'No frontmost application found') without suggesting next steps. Per pattern:recovery-guide, errors should guide LLM actions: 'No frontmost application found. Try ax_focus_app() to activate an app first.'
ax_scroll direction parameter lacks enum constraint. Description says 'up', 'down', 'left', 'right' but schema does not enforce enum. LLMs may pass invalid values like 'scroll_up' or 'vertical'.
set_directive 'type' parameter lacks enum constraint. Description lists 'reminder, habit, automation, watch' but schema does not enforce enum, inviting hallucinated values.
Transport mechanism unknown from provided source. Code shows Swift/Rust implementation but no explicit MCP server registration, HTTP listener, or STDIO setup visible. Cannot verify protocol readiness without transport confirmation.