Model Context Protocol server for web crawling powered by Crawl4AI
MCP-Crawl4AI demonstrates solid definition quality with clear naming patterns, comprehensive parameter descriptions, and well-structured schemas. All 7 tools follow verb_noun naming conventions (scrape, crawl, create_session, etc.). Input parameters are consistently documented with descriptions and constraints. However, there are notable gaps in output schema documentation, error handling guidance, and some parameter details are incomplete. The tool set is well-composed for web crawling workflows, but lacks explicit error recovery patterns and output field specifications.
Close an active crawling session and clean up associated resources
Crawl a URL and optionally follow links within a domain
Crawl multiple URLs concurrently with rate limiting
Create a new crawling session for managing state and artifacts
Retrieve a specific artifact by ID
List artifacts (captured content, files, logs) associated with a session
Scrape a single URL and extract content in a specified format
Output schemas not documented. No explicit definition of return types for any tool. LLMs cannot predict which fields to expect or how to chain tools together. Violates pattern:tool and pattern:response-shaper.
create_session and close_session lack clear descriptions of what state they manage and how artifacts relate to sessions. Session lifecycle is implicit rather than documented. This creates confusion about when sessions should be used vs direct tool calls.
No error recovery guidance. Tools lack descriptions of what errors can occur and how the agent should respond (retry, ask user, fatal). E.g., scrape and crawl can timeout, hit rate limits, encounter invalid selectors, but no guidance is provided.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 67 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 28 | - | v1 |
extraction_targets parameter in scrape tool is defined as 'array' with generic description 'Array of extraction target definitions' but lacks structure specification (items schema, required format). This forces LLMs to guess the object structure.
No pagination or result limit documentation. crawl_many accepts multiple URLs but no guidance on how many results per URL are returned or whether pagination is supported. crawl with max_pages suggests limits exist but behavior on truncation is unstated.
Tool descriptions lack guidance on prerequisites and when to use one tool vs another. E.g., when should an agent use scrape vs crawl? When is create_session necessary? Relationship between tools is implicit.