Loading ToolBench
MCP server for interacting with the arXiv API
Three tools with complete Zod schemas and descriptions. Naming follows verb_noun pattern (searchArXiv, getPaperById, getLatestPapers). Descriptions are adequate (50-100 chars) but lack actionable context for LLM selection. Parameters are typed and mostly described, but lack constraints (e.g., max_results accepts 0-100 but no validation shown). Output schema is implicit (text + resources) but not formally documented. Error handling returns text messages but lacks recovery guidance or categorization. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only operations.
Retrieve the latest papers from a specific arXiv category
Retrieve a specific paper from arXiv by its ID
Search for papers on arXiv with customizable query, sorting, and filtering options
Output schema not formally documented. Tools return {content, resources} but LLM cannot infer structure of resource objects or pagination behavior.
No tool annotations despite all tools being read-only. Missing readOnlyHint would help agents understand these are safe to call without side effects.
Error messages lack recovery guidance. 'Error searching ArXiv: <message>' tells LLM nothing about whether to retry, adjust query, or ask user. No error categorization (retryable vs user-fixable).
Parameter descriptions lack constraints. 'max_results' accepts 0-100 but no validation shown in code; LLM could pass invalid values. 'categories' accepts free-form string with no format guidance (should document arXiv category format like 'cs.AI').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
Tool descriptions too brief (50-65 chars). Do not explain WHEN to use each tool or how they differ. LLM cannot distinguish between searchArXiv (query-based) and getLatestPapers (category-based) without deeper reasoning.