Loading ToolBench
MCP server exposing the RAG pipeline as tools for AI agents. Wraps shared components (retriever, generator, eval harness) so any MCP-capable client can search the corpus, ask grounded questions, and check retrieval quality.
Three tools with basic descriptions and parameter schemas, but significant gaps in completeness. All tools have descriptions (10-20 chars baseline met), and input parameters include types and defaults. However, output schemas are undocumented, error handling is absent, and parameter descriptions lack detail on constraints, formats, and valid ranges. No tool annotations (readOnlyHint, etc.). The server exposes a functional RAG pipeline but falls short of production-grade tool design.
Answer a question grounded in the paper corpus, with source chunk IDs.
Score the retriever against the hand-labeled eval set.
Search the research-paper corpus and return the top-k chunks.
Output schemas undocumented. search_documents and ask_question return dicts with fields (chunk_id, source, text) and (answer, sources) respectively, but no schema is declared in tool definitions. LLMs cannot plan downstream calls or extract fields reliably.
Parameter descriptions lack constraints and formats. 'k' is described as 'Number of chunks to return (default 5)' but does not specify min/max bounds, valid range, or consequences of extreme values. 'query' and 'question' lack format guidance (length limits, character restrictions).
No error handling or recovery guidance. Tools do not document failure modes (e.g., Qdrant unavailable, query too long, no results found). LLMs receive no actionable error messages to guide retry or fallback behavior.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
evaluate_retrieval has empty input schema (no parameters). Description is vague ('Score the retriever against the hand-labeled eval set') and does not explain prerequisites (eval set must exist at eval/eval_set.jsonl) or output structure (mrr, per_k fields).
No tool annotations. Tools lack readOnlyHint (all three are read-only), idempotentHint (search_documents and evaluate_retrieval are idempotent), or destructiveHint. Agents cannot infer safety properties without explicit hints.