Lightweight MCP server for web scraping, search, and crawling — local-first with optional Firecrawl fallback
Four tools with clear verb-noun naming (webcrawl_scrape, webcrawl_search, webcrawl_map, webcrawl_crawl). All have descriptions (194 - 280 chars, within baseline 34 - 392 range) and input schemas with typed parameters. However, output schemas are undocumented in the source, descriptions mention return structure but no formal schema is visible. Parameter descriptions are present but minimal (10 - 50 chars vs baseline 72 chars). No error handling guidance, no enum constraints for source field, no pagination metadata (total count, next_cursor) for list-returning tools. Missing tool annotations (readOnlyHint, idempotentHint). Composition is sound: each tool has one responsibility, and names clearly distinguish them.
Crawl multiple pages starting from a URL. Uses BFS to discover and fetch pages up to max_depth links away. Respects rate limiting between requests. Args: url: Starting URL max_pages: Maximum number of pages to fetch (default: 10) max_depth: Maximum link depth from start (default: 2) include_patterns: Glob patterns for URLs to include (e.g., ["*/docs/*"]) Returns: List of {url, title, content} for each crawled page
Discover URLs on a website. Fetches the given URL and extracts all same-domain links. Args: url: Starting URL to map from limit: Maximum number of URLs to return (default: 50) Returns: List of unique same-domain URLs found on the page
Fetch a URL and extract main content as markdown. Args: url: The URL to scrape timeout: Request timeout in seconds (default: 30) Returns: Dict with: - content: markdown of the page's main content - source: one of "static_http", "static_http_retry", "firecrawl_transport_fallback", "firecrawl_quality_fallback" (see Issue #1)
Search the web using DuckDuckGo. Args: query: Search query string num_results: Maximum number of results to return (default: 5) scrape_results: If true, fetch full page content for each result (default: false) Returns: List of search results, each with url, title, snippet, and optionally content
Output schemas not formally documented. Descriptions mention return structure (e.g., 'Dict with content, source' for webcrawl_scrape; 'List of {url, title, content}' for webcrawl_crawl) but no JSON Schema is visible in source. LLMs cannot reliably parse unstructured output or plan downstream tool calls.
No pagination support. webcrawl_search and webcrawl_crawl return lists but lack limit/offset parameters and total count in response. Large result sets risk context window exhaustion; no guidance on how to fetch additional results.
No error handling guidance. Tool descriptions do not explain failure modes (network timeout, invalid URL, rate limit, blocked domain) or recovery steps. LLMs receive raw errors with no actionable next steps.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Parameter descriptions are minimal (10 - 50 chars vs baseline 72 chars). E.g., 'Request timeout in seconds (default: 30)' lacks guidance on when to adjust it or what happens on timeout. 'Glob patterns for URLs to include' does not explain glob syntax or provide examples.
No tool annotations. Tools lack readOnlyHint (all are read-only), idempotentHint (webcrawl_scrape and webcrawl_search are idempotent; webcrawl_crawl may not be if it modifies state). Annotations help LLMs reason about retry safety and side effects.