Local MCP wrapper for the LongBook Proof / RAG evaluation lab. Exposes a small allowlisted set of local tools for book chunking, indexing, retrieval, and evaluation.
Server has 11 tools with complete input schemas and descriptions. Naming follows verb_noun convention (list_, read_, chunk_, build_, retrieve, run_, report_). Descriptions are present (avg ~100 chars) but lack LLM-optimized guidance on WHEN to use each tool vs alternatives. Parameters are typed and constrained (e.g., timeout_seconds clamped 1-900, max_items 1-1000), but descriptions don't always explain the constraint rationale. Output schemas are not documented, LLMs cannot predict response structure. Error handling is minimal: no recovery guidance, no categorization of retryable vs fatal errors. Security is strong (path allowlisting, no secrets in params, blocked sensitive names), but no tool annotations (readOnlyHint/destructiveHint) to signal safety to clients.
Build a retrieval index from a book file with embeddings.
Chunk a book file into overlapping text segments and save as JSONL.
List book/manuscript files under the project.
List evaluation run directories.
List product run directories.
List report-like files under reports, outputs, or product runs.
Return read-only status for the local LongBook project.
Read a text file (book, report, or other allowed file) with size limits.
No output schemas documented. LLMs cannot predict response structure (fields, types, nesting). Tools like retrieve, run_eval, report_tables return complex results but provide no schema guidance.
Tool descriptions lack WHEN/WHY guidance. Descriptions state WHAT (e.g., 'Chunk a book file') but not WHEN to call this vs alternatives or what prerequisites exist. LLMs cannot distinguish chunk_book from build_index without trial.
No error recovery guidance. Tools invoke subprocess.run() with timeouts but return raw returncode/stdout/stderr. LLMs cannot determine if a failure is retryable, user-fixable, or fatal.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Generate report tables from evaluation results.
Retrieve relevant chunks from a built index given a query.
Run evaluation of a retrieval method against questions.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk field is present in metadata but not exposed via MCP tool annotations. Clients cannot auto-detect which tools are safe to retry.
Parameter descriptions omit constraint rationale. timeout_seconds is clamped 1-900 but description doesn't explain why (e.g., 'prevents runaway processes'). max_items clamped 1-1000 but no guidance on performance implications.