Loading ToolBench
Drive a remote BMW ISTA+ garage laptop from an MCP client. Provides screen capture, document reading, input control, and diagnostics for ISTA (BMW diagnostic software) via SSH.
ISTA-MCP has 20 tools with mostly complete schemas and descriptions, but quality is uneven. Naming is generally clear (verb_noun pattern: read_state, click_control, start_stream). Descriptions are detailed and context-rich (avg ~180 chars), exceeding the 10 - 1024 char baseline. However, 6 tools lack explicit input schemas in the visible code (read_state, read_doc, screenshot, list_controls, read_log, list_sessions, diagnose, calibrate, setup appear inferred from docstrings rather than registered schemas). Parameter descriptions are present but some lack type constraints (e.g., 'log_name' in read_log accepts free-form strings with no enum). Error handling is minimal, most tools return raw subprocess output or None without recovery guidance. Output schemas are undocumented. Security is a concern: ISTA_MCP_ALLOW_INPUT env var gates input, but no per-tool permission checks or audit logging visible.
Calibration self-test for screen capture + click accuracy. Launches a full-screen target with five numbered buttons, reads their positions, clicks each one, and compares hit positions to targets. Confirms that grab.ps1 and input.ps1 are DPI-aware and aligned.
Left-click at screen pixel coordinates (relative to grab.ps1 screenshot). Fallback for controls without reliable names; prefer click_control() for robustness.
Click a control in ISTA by name using UIAutomation (Invoke/Select patterns), with coordinate click as fallback. list_controls() shows the real names. OPT-IN via ISTA_MCP_ALLOW_INPUT=1; input only lands if ISTA runs non-elevated (Windows UIPI silently discards injected input into elevated windows).
Left double-click at screen pixel coordinates (open tree items, files). OPT-IN via ISTA_MCP_ALLOW_INPUT=1.
One-shot health check for the ISTA-MCP capture/input path. Emits compact JSON with desktop session status, Defender exclusion, pending reboot state, Task Scheduler health, ISTA process status, and frame freshness. The decisive check is scheduler_ok: capture + input both ride Windows Task Scheduler, and a pending-reboot/limbo state can leave it accepting /run and reporting success while executing NOTHING.
Input schemas not visible for 9 tools (read_state, read_doc, screenshot, list_controls, read_log, list_sessions, diagnose, calibrate, setup). Tool definitions appear inferred from docstrings rather than explicitly registered with MCPServer.tool() decorator and JSON Schema.
Output schemas completely undocumented. No tool describes what fields it returns, their types, or structure. LLMs cannot plan downstream calls or extract data reliably. E.g., read_state returns 'doc text + frame + status' but no schema; click_control returns success/failure but format unknown.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Manage the RUNASADMIN compatibility layer on ISTAGUI.exe. Check or toggle elevation status. ISTA set to always-run-as-admin means Windows UIPI silently discards every injected click/keystroke. For DIAGNOSIS run ISTA non-elevated (input works); flip it back on for PROGRAMMING/CODING sessions that need admin.
Press a special key (ENTER, TAB, ESC, etc.). OPT-IN via ISTA_MCP_ALLOW_INPUT=1.
Fast, near-live path: pull the newest frame from the continuous capture stream (C:\ista-mcp\live.jpg). No scheduler trigger, no fixed wait. Falls back to screenshot() if no stream is running.
Dump ISTA's actionable controls (buttons, tabs, list/tree items, links, fields) to show the real names for click_control(). Read-only; gives the model a menu of valid control names.
List ISTA diagnostic sessions (VIN, date, status). Reads from ISTA logs.
Right-click at screen pixel coordinates (context menus, e.g. E-Sys 'Edit FDL'). OPT-IN via ISTA_MCP_ALLOW_INPUT=1.
Read ISTA's currently displayed document (procedure, fault, functional-description) as plain text, parsed from ISTA's rendered HTML. Fastest text-only path when you don't need the screen frame.
Read ISTA or PsdzWebservice log file as text. Specify which log via the log_name parameter.
THE primary read: one round-trip returning ISTA's currently displayed document text (parsed from ISTA's own rendered HTML - full text, no OCR, no clipping) plus one screen frame and ISTA process status. Start every look-at-ISTA step here; fall back to screenshot() only for purely graphical detail (live graphs).
Reliable one-shot screen capture. Triggers the IstaGrab task, waits for a fresh frame, pulls it back. Defaults to compressed JPEG (~tens of KB); fmt='png' gives lossless ~600 KB fallback.
Mouse wheel scroll at current cursor position. OPT-IN via ISTA_MCP_ALLOW_INPUT=1.
Deploy ISTA-MCP scripts and register Windows Task Scheduler tasks on the remote garage laptop. Pushes state.ps1, grab.ps1, input.ps1, elev.ps1, diagnose.ps1, caltarget.ps1 to C:\ista-mcp and registers IstaGrab, IstaInput, IstaStream tasks. Run once after initial SSH setup.
Start a continuous capture loop in the interactive desktop session that writes the newest frame to C:\ista-mcp\live.jpg. Enables fast latest_frame() polling without per-frame scheduler overhead.
Stop the continuous capture loop. Saves hotspot bandwidth and laptop CPU.
Type text into the focused field. OPT-IN via ISTA_MCP_ALLOW_INPUT=1.
No error handling or recovery guidance. Tools return raw subprocess output, None, or exceptions without actionable next steps. E.g., _ssh() appends stderr but does not categorize errors as retryable vs fatal. LLMs cannot self-correct.
Free-form string parameters lack enums or constraints. 'log_name' in read_log accepts any string; 'key' in key() accepts any SendKeys syntax without validation. LLMs hallucinate invalid values.
No audit logging or permission checks visible. ISTA_MCP_ALLOW_INPUT env var gates input globally, but no per-tool scope declarations, per-user authorization, or call logging for compliance/debugging. Destructive tools (start_stream, calibrate, setup) lack confirmation patterns.