Loading ToolBench
MCP server for accessing arXiv API to search, retrieve, and download academic papers
The arXiv MCP server has 5 tools covering article retrieval, metadata extraction, PDF download, and content loading. Tool names follow verb-noun convention (get_, download_, load_, search_) which is correct. However, parameter descriptions are minimal (10-15 chars on average), falling significantly short of the baseline 72 chars. Input schemas are present and typed, but descriptions lack LLM-optimization guidance. Error handling returns plain strings rather than structured recovery hints. Output schemas are not documented. The load_article_to_context tool returns unbounded text content without pagination, which could exhaust context windows. Overall: solid naming and schema presence, but descriptions and output documentation are the weak points.
Download the article hosted on arXiv.org as a PDF file. This tool searches for the article based on its title, retrieves the article's PDF, and saves it to a specified download location using the arXiv ID as the filename.
Retrieve the URL of an article hosted on arXiv.org based on its title. Use this tool only for retrieving the URL. This tool searches for the article based on its title, and then fetches the corresponding URL from arXiv.org.
Retrieve information of an article hosted on arXiv.org based on its title. This tool searches for the article based on its title and retrieves arXiv ID, title, authors, link, direct PDF URL, published timestamp, last updated timestamp, and summary.
Load the article hosted on arXiv.org into context. This tool searches for the article based on its title, retrieves the article content, and loads text content into LLM context.
Performs a search query on the arXiv API based on specified parameters and returns matching article metadata. This function allows for flexible querying of the arXiv database. Only parameters that are explicitly provided will be included in the final search query. Results are returned in a JSON-formatted string with article titles as keys and their corresponding arXiv IDs as values.
Parameter descriptions are extremely brief (10-15 chars). Rubric baseline is 72 chars. LLMs cannot infer parameter semantics from 'Article title.', no format, constraints, or dependency hints provided.
Output schemas are not documented. Tools return strings (error messages, URLs, JSON) but callers cannot know what fields to expect in successful responses. get_details returns a JSON-formatted string, but the schema is not formally declared.
load_article_to_context returns unbounded PDF text content (entire article). No pagination, length limit, or warning. Can easily exceed context windows. Should implement result limiting and pagination per pattern baseline.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 54 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 50 | - | v1 |
Error handling returns plain strings without recovery guidance. E.g. 'Unable to retrieve data from arXiv.org.' does not tell the LLM what to do next, whether to retry, or if the input was invalid. Errors should follow recovery-guide pattern.
search_arxiv accepts optional 'start' parameter but does not document max_results or pagination limit. Current implementation allows arbitrary pagination.
Response field naming inconsistency: search_arxiv returns 'article titles as keys and their corresponding arXiv IDs as values' (dict structure), but other tools return different formats (URLs, JSON strings). This inconsistency forces LLMs to adapt parsing logic per tool.
get_details and search_arxiv return JSON-formatted strings rather than structured objects. LLMs must parse JSON within strings, wasting tokens and increasing error likelihood. Should return native objects with typed fields.