Documentation assistant leveraging Model Context Protocol for searching, retrieving, and downloading documentation
TheAlmanac implements 5 tools with basic structure and clear verb-noun naming conventions (search_documentation, get_documentation, etc.). However, critical gaps exist: parameter descriptions are present but often generic; output schemas are not documented anywhere in the codebase; error handling is minimal and non-instructive; parameter constraints are missing (e.g., no enum for source_name, no min/max for max_results). Tools are simple read-only operations that handle data retrieval but lack the LLM-optimization patterns needed for production agent use. Tool definitions are explicitly visible in almanac/mcp/tools.py, so they are not inferred. The per-tool average reflects adequate naming and presence of descriptions, but weak schema documentation and error guidance.
Start downloading documentation from a URL (async)
Get specific documentation content
Get the status of an asynchronous task
List available documentation sources or documents
Search documentation for relevant information
No output schemas documented. Tools return dict responses (e.g., {results, count, message}) but LLMs have no way to know what fields to expect, their types, or required structure. This breaks downstream tool chaining and forces LLMs to guess field names.
Parameter descriptions are generic and lack specificity. E.g., 'source_name': 'The specific documentation source to search, or empty for all sources', does not specify format, valid values, or examples. 'max_results': 'Maximum number of results to return', lacks min/max bounds.
No enum or constraint declarations on free-form string parameters. 'source_name' accepts any string with no validation or enumeration of valid sources. This invites hallucinated source names from LLMs.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 37 | - | v1 |
Error responses are minimal and non-instructive. E.g., {error: 'No query provided'} or {error: 'Source name and document name are required'} do not guide the LLM on what to do next or suggest alternatives. No recovery guidance.
No pagination support on search_documentation. If results can be large, the tool should accept page/offset/limit parameters and return a total count. Returning unbounded lists risks context window exhaustion.
Tool descriptions do not clarify when to use related tools. E.g., search_documentation vs get_documentation, when should the LLM choose one over the other? The distinction is not explicit.
download_documentation is marked as WRITE and is asynchronous, but no confirmation pattern or dry-run support. Agents may inadvertently trigger downloads without explicit user consent or preview.
No tool-level permission declarations (e.g., 'read:documentation', 'write:documentation'). Scope requirements are not declared, limiting audit clarity and least-privilege configuration.