Site crawler and MCP server based on crawl4ai
Single tool 'crawl_website' has a complete input schema with 6 parameters, all typed and described. Tool description is adequate (145 chars, within 10-1024 range). However, output schema is undocumented, the tool returns a plain string with no structured format specified. Parameter descriptions are present but generic; they lack actionable constraints (e.g., max_depth range, delay bounds, mode enum values). No error handling guidance, the catch-all exception handler returns a bare error message without recovery hints. Tool name 'crawl_website' is clear and verb-based, but the description does not explain WHEN to use this tool vs alternatives, or what the output structure contains. Missing: pagination support (no limit on crawled_pages), idempotency guarantees, and permission/scope declarations. The tool performs a destructive write operation (creates files) but has no dry-run or confirmation pattern.
Crawls a website recursively and saves the pages as Markdown files or explores URLs.
Output schema undocumented. Tool returns a plain string; LLMs cannot parse structured results or extract IDs for chaining. Should return {success: bool, page_count: int, output_dir: string, index_path?: string, urls_path?: string, error?: string}.
Parameter constraints missing. 'mode' accepts 'crawl' or 'explore' but is not declared as an enum. 'max_depth' and 'max_pages' lack min/max bounds. 'delay' has no range validation. Free-form strings invite hallucinated values.
Error handling provides no recovery guidance. Exception caught and returned as plain string 'Error while crawling {url}: {str(e)}'. LLM cannot determine if error is retryable, user-fixable, or fatal.
Destructive operation (file writes) lacks confirmation or dry-run pattern. Tool creates output directories and files without user confirmation. Agents may accidentally overwrite data.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
No pagination or result limits documented. Tool accepts max_pages parameter but does not explain what happens if crawl exceeds limits, or how to retrieve additional pages. Large crawls could exhaust context.