Loading ToolBench
Model-neutral computer-use runtime with grounded actions and explicit evidence
23 tools with significant quality gaps. Most tools lack proper input schemas (many inferred from code, not explicitly registered). Descriptions are present but often terse (10-50 chars), below the 50-200 char LLM-optimized baseline. Parameter descriptions are sparse or missing entirely. No documented output schemas. Multiple tools with identical or near-identical names (e.g., two 'type_text' tools at indices 4 and 20) create ambiguity. Error handling is absent, no recovery guidance, no categorization. Security concerns: tools like 'open_application' and 'click' operate on user systems without permission gates or audit trails. Composition issues: tools are tightly coupled to specific UI automation backends (Playwright, pyautogui, OmniParser) rather than abstracting a clean agent interface.
Click a UI element on the screen. Looks up the element's center coordinates from the last OmniParser scan and sends a mouse click there. Use get_elements first to see what's available.
Click a numbered element from the last inspect_screen call.
Double-click a UI element on the screen. Same as click() but sends two clicks in quick succession. Useful for opening files in editors or selecting words in text fields.
Return the list of UI elements detected in the last screen scan. Each element has an id, label (text description), x/y coordinates for its center, and its bounding box. Call this before deciding what to click so you know what's on screen.
Resolve a SoM id or '[n] label' string to a clickable point from the last scan.
Capture a display and return numbered UI elements (SoM).
Duplicate tool names: 'type_text' appears at indices 4 and 20 with different signatures (one takes 'text' only, the other takes 'element_id', 'text', 'press_enter'). LLMs cannot disambiguate and will select the wrong variant.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., element IDs from get_elements, page text from playwright_get_page_text). Violates pattern:tool and pattern:response-shaper.
Descriptions are terse (10-50 chars) and lack context. E.g., 'Scroll at the position of a screen element' does not explain WHEN to use scroll vs playwright_scroll, or what happens if the element is off-screen. Baseline is 50-200 chars for LLM-optimized descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
Launch or focus an installed macOS application and verify that it came to the front.
Click an element by its ARIA role and name using Playwright.
Click an element by its text content using Playwright.
Get the text content of the current page using Playwright.
Press a keyboard key or shortcut using Playwright.
Scroll the current browser page.
Type text into the currently focused browser control.
Validate flight-search details; return a clarification or a ready Google Flights URL.
Press a keyboard key or shortcut. Examples: press("enter") press("tab") press("escape") press("cmd+s") # save press("ctrl+shift+p") # VS Code command palette
Press a key or chord, e.g. ['cmd', 's'] or ['enter'].
Right-click a UI element to open its context menu.
Scroll at the position of a screen element.
Capture the screen and return Set-of-Marks UI elements (id, label, bbox).
Type text at the current cursor position. The text is typed character by character using pyautogui. Newlines in the string are converted to Enter keypresses. Make sure the right input field is focused before calling this.
Focus an element by id, then type. Set press_enter to submit.
Click a web_snapshot index in the attached browser.
Occlusion-pruned accessibility snapshot of the attached browser page.
No error handling or recovery guidance. Tools like 'click' and 'open_application' provide no feedback on failure (element not found, app not installed, permission denied). LLMs cannot self-correct or retry intelligently.
Security: 'open_application' and system-level tools (click, type_text via pyautogui) lack permission gates and audit trails. No indication of what permissions are required or who can invoke them. Violates pattern:permission-gate and pattern:audit-trail.