Loading ToolBench
Web scraper + validated extraction contracts for AI agents. Markdown, PDF/DOCX, JSON-LD/microdata/commerce/forms/analytics, structured errors, diagnostics, and LLM extraction to any OpenAI-compatible endpoint.
TheCrawler demonstrates solid tool design with comprehensive parameter schemas and detailed descriptions. All 9 tools have explicit input schemas with typed properties and descriptions. Tool names follow verb_noun convention (crawl, extract_*, search_*, list_*, diagnose_*, map_*). Descriptions are substantive (100-400+ chars), explaining WHAT the tool does and WHEN to use it. Error handling is structured (errorType enum, retryable flag). However, output schemas are not documented, LLMs cannot predict response structure. Some parameter descriptions lack format constraints (e.g., 'proxyUrl' should specify format). Tool composition is strong: crawl() is the core, with specialized variants (crawl_markdown, crawl_sitemap, search_and_crawl) and extraction tools (extract_structured, extract_extraction_contract) that chain well. Security: API keys are passed as parameters (llmApiKey, serpApiKey) rather than injected server-side, a significant risk. No tool annotations (readOnlyHint, destructiveHint) despite all tools being read-only.
Scrape one or more URLs. Returns rich structured data: title, description, text, markdown, links, images, meta, OG/Twitter cards, JSON-LD, microdata, headings, tables, forms, redirect chain, hreflang, pagination links, commerce data (price/rating/sku from JSON-LD), analytics tracker detection (16 trackers), and optional email-like/phone-like public text fields when explicitly enabled. Auto-handles PDF and DOCX URLs. Default Cheerio (fast HTTP+parse); set usePlaywright=true for JS rendering, or adaptiveCrawling=true to auto-detect SPAs and re-crawl them with Playwright. On failure, returns structured error with errorType (dns|timeout|rate-limit|blocked-bot|js-required|http-4xx|http-5xx|parse|network|unknown) and retryable flag.
Extract clean markdown for a single URL. Strips boilerplate (nav/header/footer/cookie banners). Returns markdown text only — no surrounding metadata. Ideal for feeding a single page to an LLM. For multi-URL or structured output, use `crawl`, `extract_structured`, or `extract_extraction_contract` instead.
Crawl all URLs listed in a sitemap.xml file. Returns structured data for each URL in the sitemap. Respects robots.txt and crawl-delay directives.
Diagnose whether a page is ready for extraction with a given contract, WITHOUT calling an LLM. Scores the page against contract requirements (required fields present, structured data quality, text length, etc.). Returns a diagnostic verdict and recommended next steps. Useful for pre-flight checks before expensive LLM calls.
Output schemas not documented. LLMs cannot predict response structure for any tool. Responses are JSON but fields, types, and nesting are undocumented.
API keys (llmApiKey, serpApiKey) exposed as tool parameters. Credentials should be injected server-side via environment variables or vault. Agent traces log all parameters, secrets leak into logs.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All 9 tools are read-only and idempotent, annotating them would help agents reason about safety and retry logic.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | <=2025-11-25 | v2 |
Extract structured data using a built-in extraction contract (e.g. real-estate-listing, product, job-posting). Contracts bundle a JSON Schema, extraction prompt, and validation rules. Returns extracted data + validation results. Requires llmBaseUrl + llmModel.
LLM-powered structured extraction from one or more URLs. Supply a JSON Schema and optional prompt. Returns extracted data validated against the schema, with per-field confidence scores and source grounding (which page text each field came from). Requires llmBaseUrl + llmModel (OpenAI-compatible endpoint).
List all built-in extraction contracts available for use with extract_extraction_contract. Returns contract names, domains, versions, descriptions, and required fields.
Discover all links from a URL. Returns a deduplicated, sorted list of discovered links with their anchor text, internal/external flags, and source URL. Useful for site mapping and link analysis.
Search Google for a query, then scrape the top N results. Returns structured data per result page. Without a SerpAPI key, falls back to scraping Google's HTML — fragile, can break when Google updates their HTML. For production agent use, supply serpApiKey.
Parameter format constraints missing. 'proxyUrl' should specify format (http://user:pass@host:port). 'cssSelector' should note CSS selector syntax. 'jsonSchema' should clarify JSON Schema draft version.
Error recovery guidance incomplete. crawl() returns structured errorType but does not suggest next steps (e.g., 'If js-required, retry with usePlaywright=true'). extract_* tools do not document LLM endpoint failure modes.