Loading ToolBench
MCP server for semantic search, document retrieval, vault browsing, indexing status, and automated indexing of an Obsidian vault with hybrid search (vector + keyword), enrichment, and multi-source support
RAG-In-A-Box presents 15 tools with consistent naming conventions (verb_noun: file_search, file_get_chunk, file_browse_documents, etc.) and comprehensive parameter schemas. However, tool descriptions are functional but generic, most lack context about WHEN to use each tool vs alternatives or prerequisites. Output schemas are not documented in visible source. Significant gaps: no error handling guidance, no enumerated constraints on parameters like 'return_mode' or 'status', no distinction between idempotent vs destructive operations in descriptions (though risks are labeled). Backward-compatibility aliases (vault_* duplicates) add cognitive load without value. Security is reasonable (no secrets in params), but composition could be tighter, 4 tools are straight duplicates (file_search/vault_search, file_get_chunk/vault_get_chunk, file_browse_documents/vault_browse_documents, file_index_status/vault_status). Average param count is 4.3, which is reasonable. Baseline expectations: descriptions ~194 chars, params ~72 chars; most tools here are 50-150 chars, shorter than baseline but not failing. Core RAG operations (search, retrieve, browse, index) are well-defined structurally but lack the LLM-optimized framing that would push this into 70+ territory.
List documents in the index with optional filtering by source, status, or folder
Delete a document from the index by rel_path, abs_path, or doc_id
Estimate embedding and LLM costs for a hypothetical indexing run given source parameters
Retrieve a specific chunk by its chunk_id with full content and metadata
Cancel the currently active indexing run if present
List recent indexing runs with status, timing, and error details
Get the current status of the document index including vector store health, FTS status, freshness metrics, disk usage, and any active indexing runs
Four tools are backward-compatibility aliases (vault_search, vault_get_chunk, vault_browse_documents, vault_status) that duplicate primary tools with reduced or identical parameter sets. This confuses agent routing, wastes reasoning cycles, and violates the single-responsibility principle. Aliases should be removed or the API versioned cleanly.
Descriptions lack LLM-optimized framing: no explanation of WHEN to use each tool vs alternatives, no dependency hints (e.g. 'call file_browse_documents first to see available sources'), and no contextual prerequisites. Most descriptions are 50-150 chars (below the 194-char baseline) and functional but generic.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | 2025-06-18+ | v2 |
Manually trigger a full index refresh across all configured sources or a targeted single-document index
Manually trigger a full-text search index rebuild for keyword search
Search the index via hybrid search (vector + keyword ranking with optional reranking). Returns document chunks with relevance scores, metadata, and optional full content.
Alias for file_browse_documents tool for backward compatibility
Alias for file_get_chunk tool for backward compatibility
Alias for file_index_update tool for backward compatibility
Alias for file_search tool for backward compatibility
Alias for file_index_status tool for backward compatibility
No enumerated constraints on parameters with discrete valid values. E.g., file_search's 'return_mode' accepts 'slim'|'compact'|'full' (documented in description), but no formal enum in schema. LLMs cannot reliably pick from text descriptions; formal enums are machine-readable and prevent hallucination.
Output schemas are not documented. LLMs cannot plan downstream calls or extract required fields (e.g. chunk_id from search results for use in file_get_chunk) without knowing response structure. Rubric baseline requires 100% of A+ tools to have documented return types.
Destructive and stateful tools (file_index_update, file_delete_document, file_index_cancel, file_rebuild_fts) lack guidance in descriptions about idempotence, side effects, or confirmation. Agents need to know: can I retry safely? Is this operation idempotent? The risk labels (WRITE, DESTRUCTIVE) exist in metadata but not in human-readable descriptions.
Mutually exclusive parameters are not documented. E.g., file_index_update accepts rel_path, abs_path, OR doc_id, the description should state 'exactly one of rel_path, abs_path, or doc_id is required; do not pass multiple.' Currently, LLMs may pass more than one, causing ambiguity.
No error handling guidance. E.g., if file_search finds 0 results, should the agent retry with different parameters, broaden the search, or report to the user? No recovery hints provided. Rubric requires error responses to guide the LLM: 'User not found. Try search_users() with a partial name.'
vault_index_update has an empty input schema (no parameters), which breaks the tool entirely, agents cannot specify what to index. This is likely a documentation error; the primary file_index_update has 5 optional params. The alias should either mirror the primary or be removed.