Loading ToolBench
Modular MCP server: see and drive the desktop. Provides persistent PipeWire capture, per-monitor crops, pointer/keyboard input via RemoteDesktop portal, optional Set-of-Marks annotation (OCR + icons), window awareness, frame reliability, session recording/replay, and self-adaptive micro-loops.
screen-mcp demonstrates solid naming (verb_noun pattern: screen_screenshot, screen_click) and comprehensive parameter schemas with types and descriptions. However, tool descriptions are inconsistent in depth, some are detailed (screen_screenshot: 127 chars), others minimal (screen_sense, screen_diag: <50 chars). Output schemas are not documented in the visible code. Error handling is present but lacks recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications. Composition is good (single responsibility per tool), but descriptions for some tools fall below the 50-200 char LLM-optimized range.
Click at a coordinate or on a named element. Supports click-by-id (from annotate), click-by-label (OCR), and raw x/y coordinates.
Diagnostic dump of live session/geo/cursor/grounding health. Reach for it first when something's off.
Drag from one coordinate to another.
Press a keyboard key or key combination (e.g., 'Return', 'ctrl+c', 'alt+Tab').
Read a whole scrollable view in one call. Scrolls down until content stops moving (or max_pages), annotating each screen and accumulating unique interactables.
Replay a recorded session.
Start recording the session (frames + actions).
Output schemas not documented. Tools return results but LLMs cannot infer response structure (fields, types, pagination). Forces agents to guess what data is available.
Minimal descriptions on diagnostic/utility tools. screen_sense (47 chars), screen_diag (48 chars), screen_reload (50 chars) fall below 50-char minimum for LLM clarity. Agents cannot determine when to invoke these vs alternatives.
No tool annotations despite explicit risk classifications. screen_click, screen_scroll, screen_drag, screen_type, screen_key, screen_reload are WRITE operations but lack destructiveHint. screen_screenshot, screen_sense, screen_diag are READ_ONLY but lack readOnlyHint. Agents cannot reason about safety/idempotency.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Stop recording and save the session.
Hot-reload: re-exec in place (os.execv) — picks up code edits and new tools (via notifications/tools/list_changed) with no /mcp reconnect.
Capture the desktop or a region, with optional annotation (OCR/OmniParser), cursor marker, and post-action screenshot.
Scroll in a direction (up/down/left/right) by a number of notches.
Ambient self-learning signal: diff this frame against the last one the agent saw and return a compact SENSE signal + human hint so it adapts without being told.
Type text via keysyms (ASCII) or clipboard paste (Unicode). Non-ASCII strings auto-paste via wl-copy + Ctrl+V.
Human-eye 1 fps observation of a region/monitor. Samples at fps (default 1.0) for seconds (default 6) and returns a timeline + verdict (settled/evolving/jitter/unstable).
Error handling lacks recovery guidance. Code shows exception handling but no visible error messages that tell LLMs 'what to do next' (e.g., 'User takeover detected. Pass force=true to regain control' or 'Coordinate out of bounds. Try screen_diag to check monitor geometry').
Parameter descriptions lack format/constraint details. 'region' param accepts [x, y, w, h] but description does not specify units (pixels?), coordinate space origin, or valid ranges. 'settle' param lacks min/max bounds (clamped 0-60000 ms?). LLMs cannot validate inputs before calling.