Loading ToolBench
RAG-powered codebase Q&A for multi-repo codebases, with LangChain and MCP server
Five tools with complete schemas and descriptions. Naming is clear and verb-driven (oracle_query, oracle_search, oracle_list_repos, oracle_expand, oracle_reindex). Descriptions are detailed (100-300 chars) and explain WHEN to use each tool. All parameters have types and descriptions. However, oracle_search's description is verbose (>500 chars) and includes implementation details (SQLite scan, over-fetch capping) that distract from user intent. Error handling is minimal, no recovery guidance or actionable error messages visible. Output schemas are not formally documented in the code; responses are inferred from implementation. oracle_reindex lacks confirmation/dry-run pattern for a destructive operation.
Read a window of lines around a specific position in an indexed file. Use after oracle_search to see the context around a chunk without leaving the oracle. Reads the file from disk via the indexed absolutePath; if the working copy has changed since indexing, the lines may not match what oracle_search returned — check oracle_list_repos for the indexed timestamp.
List repos actually present in the vector index, with chunk and file counts. Reflects what oracle_search / oracle_query can answer over — not just what exists on disk.
Ask a natural-language question about the indexed codebase. Returns an LLM-generated answer with source citations. Use this for understanding code, finding implementations, or learning how systems connect across repos.
Run the indexing pipeline against the configured scan root. Incremental: only changed and new files are re-embedded, deleted files are pruned from the store. Use this after merging a PR you want the oracle to see immediately, instead of waiting for the next scheduled reindex.
Raw vector similarity search over the indexed codebase. Returns matching code/doc chunks with metadata. Use this when you need specific code snippets rather than an interpreted answer. No LLM involved — pure embedding retrieval.
oracle_search description is 500+ chars and includes implementation details (SQLite scan, over-fetch capping) that distract from user intent. Should focus on WHAT the tool does and WHEN to use it, not HOW the backend works.
No error handling guidance visible. Tools do not return actionable error messages (e.g., 'Repo not found. Available repos: [list]'). LLMs cannot self-correct without recovery hints.
oracle_reindex is a destructive operation (WRITE risk) with no confirmation step, dry-run option, or idempotent hint. Agents could trigger unintended reindexing without safeguards.
Output schemas are not formally documented in code. Responses are inferred from implementation (e.g., oracle_query returns {content: [{type, text}]}). LLMs cannot plan downstream tool calls without explicit schema documentation.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
oracle_search's path_glob parameter description is 300+ chars and mixes semantics (picomatch syntax, AND-composition, over-fetch capping). Should separate constraint documentation from usage guidance.