Loading ToolBench
A Model Context Protocol (MCP) server for Crawl4AI that enables web crawling and content extraction as structured markdown
The server has a single tool 'crawl' with a well-structured schema and comprehensive documentation. The tool name is clear and verb-based. The description is detailed and includes helpful performance warnings and optimization tips. However, there are gaps in parameter descriptions (some lack clarity on format/constraints), no output schema documentation, and minimal error handling guidance. The schema is well-typed with defaults, but several parameters could benefit from tighter constraints and format specifications. Overall, the tool shows good intent but falls short of production-grade clarity in several areas.
Crawls a website and saves its content as structured markdown to a file. ⚠️ PERFORMANCE WARNING: This tool can take from 30 seconds to several minutes depending on the site. Heavy/SPA sites (React, Next.js, Mintlify), high `max_depth`, and the first crawl of a session (Playwright browser startup) are especially slow. The MCP client timeout should be set generously (e.g. 600000 ms / 10 min). TIPS to speed up crawls: - Use `css_selector` to extract only the relevant content (e.g. 'main', 'article'). - Use `wait_for_selector` for single-page applications. - Lower `max_depth` (1 = single page) when you don't need recursive crawling. - Use `max_pages` (e.g. 1) to cap the number of pages crawled. `max_pages=1` fetches exactly one page WITHOUT following any link - the safe way to grab a single doc page, and it preserves `<a>` text in the output (never strip anchors from the DOM to prevent link fan-out: that silently deletes every linked term, e.g. type names in API docs). - Warn the user before launching a crawl that it may take a while. - Custom JavaScript code (js_code) requires CRAWL4AI_MCP_ALLOW_JS=true environment variable.
Output schema is not documented. The tool returns a complex string containing markdown, file paths, stats, and links, but no structured output schema is provided to guide LLM downstream reasoning. LLMs cannot reliably extract fields like file_path, stats, or links without explicit field documentation.
Parameter descriptions lack actionable constraints. For example: 'max_depth' has no explanation of what crawling means or what depth 1 vs 2 vs 5 produces. 'css_selector' lacks format hints or examples. 'delay_before_return_html' has no units specified (seconds is in description, but units should be explicit in the param). 'js_code' mentions an environment variable but does not state what valid JavaScript looks like or constraints on execution.
Error handling is minimal and non-actionable. The tool catches exceptions and returns generic error strings like 'Error: {error}'. There is no categorization (retryable vs fatal), no recovery guidance, and no structured error schema. An LLM encountering an error has no guidance on whether to retry, adjust parameters, or give up.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 63 | 2026-07-28+ | v2 |
| 2026-04-07 | F | 23 | - | v1 |
No input validation or constraint enforcement visible in the tool. Parameters like max_depth, max_pages, and verbose accept any type without runtime validation. LLMs are known to pass absurd values (max_depth=999999). The tool should validate inputs early and return clear, actionable errors.
The 'session_id' parameter is documented as 'Session ID for maintaining crawl state' but it is unclear whether this is an opaque token the agent must track, or a user-facing session name. No guidance is provided on how sessions work, when to reuse them, or what state they maintain. This forces LLMs to guess.
The truncation of content at 50000 characters is mentioned in the code but not documented in the tool description. LLMs do not know they will receive partial results, potentially causing them to assume they have complete data and make incorrect decisions downstream.