MCP server for searching and extracting academic papers from arXiv with support for multiple transport types (stdio, SSE, streamable HTTP)
Critical issues: Only 2 unique tools exist (search_papers, extract_info), but the tool list shows 6 duplicates, indicating either a registration bug or inference error. Both tools have basic descriptions (60-80 chars, below the 194-char baseline), but lack parameter descriptions entirely. Input schemas are present with type info, but parameter descriptions are missing. No output schema documentation. No error handling guidance. No tool annotations. The server lacks composition patterns, search_papers writes to disk but extract_info only reads, yet they're not clearly distinguished in descriptions. No pagination support despite potentially returning multiple papers.
Search for information about a specific paper across all topic directories.
Search for information about a specific paper across all topic directories.
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a topic and store their information.
Search for papers on arXiv based on a topic and store their information.
Search for papers on arXiv based on a topic and store their information.
Duplicate tool registration: search_papers and extract_info each appear 3 times in the tool list. This suggests either a registration bug or the evaluator inferred duplicates. If real, the server will expose the same tool multiple times, confusing LLMs. If inferred, the actual tool count is 2, not 6.
Missing parameter descriptions: Both tools have input schemas with type info but zero descriptions for 'topic', 'max_results', and 'paper_id'. LLMs cannot infer parameter meaning from names alone, 'topic' could mean search query, category, or subject area. Descriptions are required per pattern:tool-description.
No output schema documentation: Tools return List[str] (paper IDs) and str (JSON or error message), but no structured output schema is documented. LLMs cannot plan downstream calls or extract fields without knowing the response structure. Baseline: 100% of A+ tools document return types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2025-06-18+ | v2 |
No error handling guidance: extract_info returns a bare string 'There's no saved information related to paper {paper_id}.' without suggesting recovery steps (e.g., 'Try search_papers() first'). Per pattern:recovery-guide, errors must tell the LLM what to do next.
No pagination or result limits: search_papers accepts max_results but does not document the default (5) clearly in the parameter description, and extract_info has no limit. Returning unbounded lists risks context window exhaustion. Baseline: tools returning lists should accept limit/offset and return total count.