Loading ToolBench
A web crawler MCP server that crawls web pages and extracts content with support for robots.txt compliance and concurrent requests
Single tool 'crawl' has a basic schema with type and format declarations, but critical gaps undermine quality. Description is minimal (10 chars: 'Crawl a web page and extract content'), below the 34-char p10 baseline and lacks WHEN/WHY context. Parameters lack descriptions: 'depth' has only 'Maximum crawl depth' (22 chars, below 72-char baseline). No output schema documented, LLMs cannot predict return structure. No error guidance (e.g., 'If robots.txt blocks, try a different URL'). Tool does not declare idempotency or side effects. No pagination support despite crawling potentially returning large link lists. Naming 'crawl' is generic; 'crawl_url' or 'fetch_page_content' would be clearer.
Crawl a web page and extract content
Tool description is 10 characters, far below 34-char p10 baseline. Lacks WHEN to use, prerequisites, and side effects. LLMs cannot determine if this tool is appropriate for their task.
No output schema documented. Tool returns JSON with 'url', 'content', 'links' fields, but LLMs have no formal specification of return types, field names, or structure. Prevents downstream tool chaining.
Parameter 'depth' lacks meaningful description. Current text 'Maximum crawl depth' (22 chars) does not explain units, default behavior, or interaction with config.maxDepth. Should state: 'Maximum crawl depth (0-N levels; default 0 = single page only). Overrides server MAX_DEPTH if lower.'
No error guidance. Code catches errors and returns 'Crawl failed: <message>' but does not guide LLM on recovery. Should return: 'robots.txt blocks this URL. Try a different domain or contact the site owner.' or 'Timeout after 5s. URL may be slow or unreachable. Retry with a longer timeout or skip.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 34 | 2026-07-28+ | v2 |
No pagination or result limits documented. Tool can return unbounded link lists (all links on a page). Large crawls risk context window exhaustion. Should cap results (e.g., 'Returns up to 50 links') and document in description.