Loading ToolBench
MCP server for searching and retrieving arXiv papers with PDF download capabilities
The arXiv server has two tools with reasonable structure but significant gaps in descriptions, parameter documentation, and output schema clarity. Both tools have action-verb names and acceptable descriptions (>20 chars), but parameter descriptions are sparse or missing entirely. The schema is partially visible but output schemas are not documented. Error handling exists but lacks recovery guidance. The server follows basic patterns but falls short of production-grade quality.
Download the PDF content of a specific arXiv paper and optionally save it to disk or analyze with Claude vision
Search for papers on arXiv using either search query or paper IDs. Supports both keyword search and direct ID lookups with pagination and sorting options.
Output schemas not documented. Neither tool documents what fields are returned or their types. The code defines ArxivPaper model, but it's not exposed in tool definitions. LLMs cannot plan downstream operations without knowing what data to extract.
Error handling lacks recovery guidance. Code raises ToolError in some paths (e.g., XML parsing, invalid input) but no error messages are visible that guide LLM recovery. Users/LLMs see a bare exception without 'Try X next' context.
Parameter descriptions are sparse or generic. 'save_path' is 'Optional local file path to save the PDF', does not clarify absolute vs. relative paths, overwrite behavior, or what happens if the directory doesn't exist. 'max_results' has no guidance on reasonable bounds (should it be 1-1000? 1-100?).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 44 | 2026-07-28+ | v2 |
Tool descriptions lack WHEN context. search_papers says what it does but not when to call it (e.g., 'Use this for exploratory search; if you have a specific paper ID, use get_paper_pdf'). This forces LLMs to guess between tools.
get_paper_pdf's analyze_with_claude parameter is ambiguous. Description mentions it 'requires OPENAI_API_KEY' but does not explain what the tool returns when analyze_with_claude=true. Does it return Claude's analysis? The raw PDF content? Both? This creates ambiguity in LLM tool selection.
Mutually exclusive parameters (search_query vs id_list) are validated in model_post_init, which is good, but error messages are not visible in the schema/description. The LLM will only learn this constraint after a failure, suboptimal for planning.
No pagination metadata in documented output. search_papers accepts start and max_results parameters, suggesting paginated results, but the tool definition doesn't document what the total count is or how to fetch the next page. Code doesn't show a 'next_cursor' or 'total_count' being returned.
analyze_with_claude is a boolean flag that triggers a different code path (vision analysis). The tool description should clarify that when true, it returns analysis text rather than PDF content, and that the response structure differs.