Loading ToolBench
Crawl a documentation site with Firecrawl and expose it as a focused MCP server.
Four tools with complete schemas and descriptions. Naming follows verb_noun convention (search_, get_, list_, refresh_). Descriptions are adequate (50-150 chars) but lack LLM-optimized guidance on when to use each tool vs. alternatives. Parameters have types and descriptions. Output schemas are documented in code but not formally declared in tool registration. Error handling includes recovery guidance (e.g., 'Run refresh_docs' vs. 'Run reindex'), which is excellent. Missing: tool annotations (readOnlyHint/destructiveHint), explicit output schema documentation in tool definitions, and parameter constraints (enums, ranges). The refresh_docs tool lacks a confirmation/dry-run pattern despite being destructive.
Return one captured page: metadata, source URL, and full markdown.
List captured pages, optionally filtered to one section or category.
Re-crawl the source site and rebuild the local corpus and index.
Search the captured docs and return ranked matching pages.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint). refresh_docs is destructive but unmarked; search_docs, get_doc, list_docs are read-only but unmarked. LLMs cannot infer safety properties without explicit hints.
Output schemas not formally documented in tool definitions. Code shows return dicts with fields (e.g., 'results', 'error', 'next_actions'), but fastmcp tool registration does not declare output schema. LLMs cannot plan downstream calls without knowing response structure.
refresh_docs (destructive operation) lacks confirmation/dry-run pattern. No way for an agent to preview changes before committing. Agents can accidentally re-crawl thousands of pages and incur Firecrawl costs.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
Parameter 'limit' in search_docs and list_docs lacks range constraints. No min/max documented. LLMs could pass limit=999999, causing performance issues or timeouts.
Descriptions do not explain when to use each tool vs. alternatives. E.g., search_docs vs. list_docs distinction is unclear. LLMs may pick the wrong tool when both seem applicable.