Loading ToolBench
Local macOS sandbox runner for disposable Tart-backed VM testing
macbox provides 38 tools with consistent naming (verb_noun pattern) and descriptions for all tools. However, parameter descriptions are minimal (often just 1-2 words), schemas lack detail (no enums, ranges, or format constraints), and output schemas are entirely undocumented. Error handling is absent, tools return raw CLI output without recovery guidance. Security concerns: no validation of vm_name or file paths visible in tool code, and destructive operations (destroy_sandbox) lack confirmation patterns. Composition is reasonable (single responsibility per tool), but many tools lack the LLM-optimized descriptions (50-200 chars) and parameter constraints needed for reliable agent use.
Assert that an app is running in the guest.
Assert that a window contains specific text or UI elements.
Click at guest screen coordinates (mouse automation via CGEvent).
Click a UI element by accessibility identifier or text.
Collect crash reports from the guest.
Collect system logs from the guest.
Create and start a disposable sandbox VM from a base image or profile.
Output schemas completely undocumented. Tools return raw CLI JSON (dict[str, Any]) with no field definitions, types, or structure guidance. LLMs cannot plan downstream calls or extract required data reliably.
Parameter descriptions are minimal (1-2 words: 'Name of the VM', 'Text to type'). No constraints, ranges, formats, or enums. LLMs cannot validate input or self-correct invalid values. Baseline: A+ tools average 72 chars per param description; these average ~20 chars.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Create a warm sandbox VM that stays running between tests.
Destroy a sandbox VM permanently.
Download a file from the guest.
Drag from one point to another on the guest screen.
Run a shell command inside the guest VM. This executes only inside the sandbox, not on the host.
Get a summary report of the current run.
Inspect the accessibility tree of the guest UI.
List running processes in the guest.
List open windows in the guest.
List local Tart VM images.
List built-in and configured sandbox profiles.
Check macbox and Tart readiness.
Open an app inside the guest VM with optional launch arguments.
Paste text into the frontmost guest app (clipboard automation).
Reset a sandbox VM back to a clean state.
Reset a warm sandbox VM back to a clean state.
Run a smoke test on an app: launch, verify it runs, take screenshot, check for crashes.
Run AppleScript inside the guest VM for UI automation or inspection.
Run diagnostic checks on macbox and Tart setup.
Run a JavaScript for Automation (JXA) script inside the guest VM. JXA reaches the ObjC bridge (Quartz/CoreGraphics) for capabilities plain AppleScript cannot express, such as synthesizing low-level input.
Upload a local .app to a warm VM and run the smoke test without recreating the sandbox.
Scroll at a point on the guest screen.
Send a key or key-combination to the guest. ``key`` is a single character, a named key (return, tab, escape, space, delete, arrows, f1-f12, etc.), or "key:<code>" for a raw virtual key code. ``modifiers`` accepts command/option/control/shift/fn (and aliases).
Stop a running sandbox VM.
Capture a screenshot of the guest desktop.
Type literal text into the frontmost guest app (keyboard automation).
Upload a local .app bundle to the guest desktop.
Upload and mount a .dmg file in the guest.
Upload an arbitrary file to the guest.
Upload and install a .pkg installer in the guest.
Watch a sandbox for changes and events.
No error handling or recovery guidance. Tools delegate to CLI and return raw output. If a command fails, LLMs receive unstructured stderr with no actionable next steps. Missing: error categorization (retryable vs fatal), invalid value feedback, and recovery hints.
Destructive operations (destroy_sandbox, reset_sandbox) lack confirmation or dry-run patterns. Agents can permanently delete VMs without user approval. No idempotency guarantees documented.
Input validation visible in code (validate_vm_name, validate_upload_path) but not reflected in parameter descriptions or schemas. LLMs have no way to know constraints (e.g. vm_name format, allowed file extensions) before calling tools.