Benchmark for Code RAG MCP servers — retrieval metrics + end-to-end agent A/B (Claude Code with vs without code intelligence)
Two tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. rag_index has a single required parameter with minimal description; code_search has two parameters but lacks detail on output structure. Neither tool description explains error handling, prerequisites, or recovery paths. Descriptions are adequate length (60-80 chars) but lack actionable context for LLM selection. No tool annotations (readOnlyHint/destructiveHint). Parameter descriptions are sparse, 'Search query' and 'Number of top results' lack format constraints, valid ranges, or examples of what constitutes a good query. Output schemas are not documented in the tool definitions, forcing LLMs to infer result structure. Risk annotations (WRITE, READ_ONLY) are present but not integrated into descriptions.
Search indexed code using BM25Okapi scoring with a whitespace tokenizer.
Walk a directory of source-code files, chunk code by function/class boundaries via tree-sitter, tokenize with whitespace tokenizer, and build a BM25Okapi index.
Output schemas not documented. LLMs cannot infer what rag_index returns (success/failure? file count? index size?) or what code_search returns (ranked results? scores? file paths?). Forces agents to guess structure.
Parameter descriptions lack actionable constraints. 'Search query' does not specify: minimum length, tokenization rules, supported operators, or what makes a query effective. 'top_k' lacks range (1 - 100? 1 - 1000?). LLMs will pass arbitrary values.
rag_index description does not explain side effects or state changes. Does it overwrite existing indices? Create new ones? Append? LLMs need to know if this is idempotent or destructive.
No error handling guidance. If rag_index fails (invalid path, permission denied, out of memory), what should the LLM do? If code_search returns no results, is that an error or valid? No recovery hints provided.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
Missing tool annotations. rag_index is marked WRITE but has no destructiveHint in schema. code_search is READ_ONLY but has no readOnlyHint. Annotations enable safer agent planning.