Loading ToolBench
BrowserAI server has 6 tools with basic schemas and descriptions, but multiple critical gaps prevent a higher score. All tools ARE explicitly registered with Zod schemas and descriptions visible in server.js, avoiding the inference penalty. However: (1) Tool names violate composition patterns, 'interact_and_extract_in_session' combines interaction AND extraction, violating single-responsibility principle. (2) Parameter descriptions are present but sparse, most lack constraints (ranges, formats, allowed values). E.g., 'timeout' in wait_for_element has no min/max bounds, 'instruction' params accept free-form strings with no guidance on format. (3) Output schemas are completely undocumented, no tool describes what fields it returns or the structure of the response. Code shows JSON.stringify({executionId, result}) but 'result' is opaque. (4) Error handling is minimal, errors are caught but not categorized or actionable. (5) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite tools having clear side-effect patterns (start_new_session, interact_and_extract_in_session, navigate_to_url are WRITE; others are READ).
Extract specific data from the current page in a browser session. Provide an array of extraction instructions, e.g., ["Extract all product names and prices as JSON array", "Get the page title and meta description"].
Check the current status and information of a browser session. Useful for debugging or verifying session state.
Interact with elements in an existing browser session. Provide an array of instructions like ["Click the login button", "Fill email field with [email protected]", "Scroll down"]. Returns updated page data after the interactions.
Navigate to a specific URL in an existing browser session.
Start a new browser session. Provide an instruction like "Go to https://example.com" or "Search for products on Amazon". Returns executionId for the session and initial page data with interactive elements and HTML markup.
No output schemas documented for any tool. Tools return JSON but structure is opaque. LLMs cannot plan downstream operations when response shape is unknown.
Tool name 'interact_and_extract_in_session' contains 'and', violating single-responsibility principle. Combines two distinct operations (interaction + extraction) that should be separate tools. Also creates redundancy with existing 'extract_from_session' tool.
Parameter 'instruction' across multiple tools is free-form string with no format guidance, constraints, or examples. LLMs will guess at format and may pass invalid or ineffective instructions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Wait for a specific element to appear on the page before proceeding. Useful for dynamic content that loads after page load. Provide element selector or description to wait for.
Numeric parameters lack min/max bounds. 'timeout' in wait_for_element has default 30 but no constraints; 'waitTime' in interact_and_extract_in_session could be negative or 1000+. Unbounded values invite LLM mistakes and API timeouts.
Error handling is minimal. Errors are caught but not categorized as retryable, user-fixable, or fatal. No recovery guidance provided to LLM (e.g., 'Try with a different selector' or 'Session expired, call start_new_session').
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear write vs read semantics. start_new_session, interact_and_extract_in_session, and navigate_to_url modify state and should be marked destructive/non-idempotent.
Description for 'interact_and_extract_in_session' says 'array of instructions' but schema defines single 'instruction' string. Mismatch causes LLM confusion about input format.
Parameter 'geoLocation' in start_new_session has only 'country' property documented, but no enum or validation. LLM could pass invalid ISO codes; missing other potential geo fields (lat/lon, timezone, etc.).