A RAG that answers RAG-stack questions — every claim backed by rag-retriever-bench measurements, served over MCP or CLI.
Three tools with complete schemas and Japanese descriptions, but descriptions are domain-specific and lack LLM-optimized guidance. All tools are READ_ONLY and well-scoped. Naming is clear (advise, compare_backends, list_traps). Parameters have types and defaults. However, descriptions lack actionable context for LLM selection (when to call each tool, what to do with results). Error handling in advise() returns structured errors but other tools lack recovery guidance. Output schemas are implicit rather than documented.
RAG検索バックエンド選定・チューニングの質問に、実測ベースの根拠を返す。 返り値の evidence は rag-retriever-bench の実測レコードと運用ノートの チャンク。回答はこの evidence の範囲で合成し、数値には source を添えること。 失敗時は {"error": ..., "hint": ...} を返す (例外は投げない)。
指定コーパス規模の全バックエンド実測値を比較表として返す。 corpus_size は 10000 か 100000。品質 (recall/ndcg/mrr@10)、レイテンシ (p50/p95)、書き込み時間、インデックス使用検証の結果を含む。
実際に踏んだ運用の罠一覧を返す。 backend 指定で絞り込み (pgvector/clickhouse/qdrant/weaviate/milvus/chroma/lancedb)。
Descriptions lack LLM-optimized guidance on WHEN to call each tool and HOW to use results. Descriptions are domain-specific (Japanese) but do not explain tool selection criteria or dependencies between tools.
Output schemas are not formally documented. advise() returns {question, evidence, note}, compare_backends() returns {corpus_size, dataset, server_backends, embedded_backends, caveat}, and list_traps() returns {backend_filter, traps}, but these are implicit in code, not in tool schema definitions.
compare_backends() and list_traps() lack error recovery guidance. compare_backends() returns {error, available} on invalid corpus_size, but does not guide the LLM on what to do next. list_traps() has no error handling.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Parameter descriptions are minimal. 'top_k' is described as 'Number of evidence chunks to retrieve' but lacks guidance on typical values or impact. 'corpus_size' states valid values (10000, 100000) but does not explain why these are the only options or what to do if neither matches the user's need.
Tool composition is unclear. The relationship between advise() (retrieval-based) and compare_backends() (tabular comparison) is not documented. An LLM cannot infer when to call compare_backends() first vs. advise() for a given question.