Loading ToolBench
Semantic search over Obsidian vault notes and textbooks via LanceDB + Ollama embeddings, with graph-based relation extraction and Claude chat integration.
The Vault Search MCP server provides 5 tools with basic naming and schemas, but has critical gaps in description quality, parameter documentation, and error handling. Tool names follow verb_noun convention (vault_search, vault_similar, vault_stats, textbook_search, graph_query), which is good. However, descriptions are minimal (all under 100 chars) and lack context about when to use each tool vs. others. All 5 tools have input schemas with type information, but parameter descriptions are sparse. No evidence of output schema documentation, error handling guidance, or security hardening. The server is STDIO-only (not remotely accessible) and provides no recovery guidance for failures.
Query the note relationship graph
Semantic search across indexed textbooks
Semantic search across all vault notes
Find notes similar to a given note
Show index statistics
Minimal tool descriptions (all <100 chars) lack context about when to use each tool, prerequisites, or what the tool returns. Descriptions must answer: What does it do? When to use it instead of similar tools? What structure is returned?
Parameter descriptions are under-specified. 'limit' parameters lack min/max constraints (e.g., is 1-1000 valid? Is the default 10 or 20?). 'query' and 'note' parameters lack format or length guidance. LLMs cannot infer these constraints from the name alone.
No evidence of output schema documentation in the tool definitions. LLMs need to know what fields are returned (e.g., does vault_search return a 'score' field? Is 'text' or 'content' the field name for note body?). This forces LLMs to guess downstream field names.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | <=2025-11-25 | v2 |
No error handling guidance in tool definitions. When vault_search returns zero results, when graph_query fails because a note does not exist, or when the embedding model is offline, LLMs have no recovery path. Error responses should guide the agent to call a discovery tool or retry.
Tool descriptions do not distinguish between vault_search and vault_similar. Both perform semantic search; the distinction is 'across all vault notes' vs. 'similar to a given note,' but a 20-char description does not make this clear enough for an LLM to confidently pick the right one on first try.
No evidence of input validation guidance. Parameter descriptions lack format constraints (e.g., is 'note' a full file path, a note name without .md, or a regex?). Free-form 'query' strings could be SQL injection vectors if not sanitized internally.