A Retrieval-Augmented Generation (RAG) system using Hacker News as a knowledge source, with MCP server integration, hybrid search capabilities, and evaluation framework
Single tool 'search' has a clear verb-noun name and Japanese description, but critical gaps undermine quality. The description (Japanese text ~180 chars) is present but lacks English translation for non-Japanese speakers, reducing accessibility. Input schema is visible with 'query' (string) and 'top_k' (integer, default 5), but 'top_k' lacks a description in the schema definition shown. Output schema is not documented, the code returns 'list[dict]' with fields 'title / url / similarity / text / rerank_score', but this structure is only mentioned in the docstring, not in a formal schema. No error handling guidance (what if index is empty? what if query is malformed?). No parameter constraints (e.g., max top_k value, query length limits). The tool is read-only (good), but no explicit idempotent or readonly annotations in the schema. Baseline: 194 chars avg description (this one ~180), 72 chars avg param annotation (none visible for top_k). Missing: output schema documentation, parameter constraints, error recovery guidance.
HN(Hacker News)の索引から、質問に近いチャンクを検索して返す。 既定では cross-encoder リランカーで再採点して返す(評価で全問1位=最良)。 RAG_RERANK_ENABLED=0 で純ベクトル検索に切替(rerank モデルの読込/推論を省く)。
Output schema not documented. Code returns list[dict] with fields title/url/similarity/text/rerank_score, but no formal schema definition visible. LLMs cannot plan downstream operations without knowing return structure.
Parameter 'top_k' lacks description in schema. Only default value (5) is shown; no guidance on valid range, max value, or when to adjust it. Baseline: 100% of A+ tools have param descriptions.
No error handling guidance. Docstring mentions 'if index is empty, return empty list', but no recovery hints for malformed queries, missing index, or timeout scenarios. LLM has no actionable next step on failure.
Description is in Japanese only. Non-Japanese speakers cannot understand tool purpose or when to invoke it. English translation or bilingual description required for accessibility.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | <=2025-11-25 | v2 |
No parameter constraints documented. 'query' has no length limits; 'top_k' has no min/max bounds. Unbounded inputs invite LLM to pass absurd values (e.g., top_k=10000) that break performance or API limits.