Self-hosted web search and guarded page extraction service with MCP server bridge
Scrapyard provides 5 well-named, read-only tools with comprehensive descriptions (avg 180 chars) and complete JSON Schema input definitions. All parameters are typed and described. Tool names follow verb_noun convention (search, extract, map, research, status). Schemas include proper constraints (enums, min/max, length limits). However, output schemas are not formally documented in the code, responses are described in prose but lack structured schema definitions. Error handling is present but generic; recovery guidance is limited. Tool annotations are declared but not applied to individual tools. No pagination support despite tools returning lists.
Fetch one public URL and return clean readable markdown. This tool extracts the main content from a single HTML page or PDF, stripping navigation, ads, and boilerplate. It preserves metadata (title, author, publish date, site name) and returns the content as markdown. Extraction uses static parsing by default; set `render=true` to enable browser rendering for JavaScript-heavy sites. The response includes `method` to show which extraction technique succeeded (trafilatura, playwright+trafilatura, or pymupdf).
Discover and return a site's sitemap URLs. This tool fetches robots.txt and common sitemap locations, parses XML sitemaps, and returns a deduplicated list of discovered URLs. It is useful for understanding a site's structure or seeding bulk extraction. The response includes `source` (robots.txt, sitemap.xml, etc.) and `count` of unique URLs found.
Run a bounded search-plus-extract research bundle. This tool combines search and extraction: it searches for the query, selects the top N results (one per domain), extracts content from each, and returns a bundle with the full search results and extracted sources. Useful for gathering diverse perspectives on a topic. Set `render=true` to enable browser rendering; `char_limit` caps content per source.
Search the live web or news index and return ranked URLs, titles, and snippets. Prefer this before extraction when you do not already know the target URL. The response preserves `engine_used` and `adapter_used` so you can report where results came from; `cached` says whether Scrapyard reused a prior response. This tool does not extract page bodies.
Output schemas not formally documented. Descriptions mention response fields (engine_used, cached, method, source, count) but no structured schema definition is visible in code. LLMs cannot reliably parse or chain outputs without explicit field documentation.
Tool annotations declared at module level (READ_ONLY_OPEN_WEB, READ_ONLY_LOCAL) but not applied to individual tool registrations. Annotations should be passed to each tool's register() call to signal read_only_hint and open_world_hint to clients.
No pagination support. scrapyard_search returns up to 20 results, scrapyard_map up to 5000 URLs, but no cursor, offset, or next_page field is documented. Large result sets risk context window exhaustion.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | <=2025-11-25 | v2 |
Inspect Scrapyard service health and configuration. Returns the service version, update check status, and search engine availability. Use this to diagnose connectivity, token, or engine issues before retrying a failed tool.
Error responses use generic codes (token_missing, service_unreachable, invalid_request) with action guidance in prose. No structured error classification (retryable vs user-fixable vs fatal) to guide agent recovery logic.
scrapyard_research combines search + extraction in one tool. While convenient, it prevents agents from composing search and extraction independently. Consider offering both bundled and separate variants.