Loading ToolBench
MCP server for searching, extracting information about, and downloading papers from arXiv
The arxiv-mcp server has 3 tools with explicit FastMCP decorators and visible source code. All tool names start with action verbs (search_, extract_, download_), which is good. However, descriptions are generic and lack actionable context about when to use each tool, parameter constraints are minimal, and critical security issues undermine the overall quality. Output schemas are not documented, code returns strings but doesn't declare structured fields. Error handling is present but not guidance-oriented. The server reads and writes files to disk with minimal input validation, creating a path traversal vulnerability.
Download the PDF of a paper given its paper_id and save it in the corresponding topic directory.
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a topic and store their information.
Path traversal vulnerability: download_path parameter is user-controlled and passed directly to os.path.join() without sanitization. Attacker can pass '../../etc/passwd' to write files outside the intended directory.
No input validation on topic parameter in search_papers. User can pass arbitrary strings that are passed directly to arxiv.Search(). No length limits, no regex constraints, no enumeration. Could cause API abuse or unexpected behavior.
Descriptions are too brief and lack context. search_papers: 88 chars (below 100-char minimum for clarity). extract_info: 78 chars. download_pdf: 119 chars. None explain WHEN to use the tool, WHAT the return value contains, or HOW to chain with other tools. Descriptions do not meet the 10-1024 character guideline with sufficient specificity.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 43 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 36 | - | v1 |
Output schemas not documented. search_papers returns List[str] (paper IDs) but no documentation of what each ID format is or how to use it downstream. extract_info returns a JSON string but the schema of the inner object is not declared. download_pdf returns a string status message. LLMs cannot reason about downstream tool chaining without documented output structure.
No pagination support in search_papers. Returns up to max_results items but no total_count, next_cursor, or offset. If an agent loops calling search_papers with different topics, there is no way to paginate through large result sets. Scales poorly.
Error handling is bare. extract_info returns 'There's no saved information related to paper {paper_id}.' and download_pdf returns 'Paper ID {paper_id} not found in any topic directory.' These are user-facing strings, not guidance for the LLM. No recovery hints, no suggestions for available alternatives, no classification (retryable vs. fatal).
No constraints on max_results parameter. Accepts any integer; no documented minimum/maximum. LLM could pass 10000 or -1. Should enforce 1 - 100 range with clear description.
Tool does not declare destructive/write intent clearly. search_papers writes JSON to disk (side effect). download_pdf downloads and writes PDFs (destructive). Descriptions do not state 'modifies state' or 'creates files', so LLMs cannot determine retry safety or compensation strategies.