Loading ToolBench
A Model Context Protocol (MCP) server that provides access to arXiv papers. Supports searching papers, getting metadata, and extracting paper content.
Server has 4 tools with complete schemas and descriptions, but quality is uneven. All tools follow verb_noun naming (search_, get_) and have non-empty descriptions (avg ~80 chars). Schemas are present with proper types and required fields. However, descriptions are terse (under 100 chars each), lack WHEN/WHY guidance, and omit error recovery hints. Parameters lack format constraints and validation guidance. Output schemas are undocumented, LLMs cannot predict response structure. No tool annotations (readOnlyHint, etc.). Error handling returns generic 'Error: {str(e)}' with no recovery guidance. Missing idempotency and chaining metadata in responses.
Extract and return the text content of a paper's PDF
Get detailed information about a specific paper by arXiv ID
Search for papers on arXiv by query, author, category, etc.
Get a summary of a paper including abstract and key information
Output schemas undocumented. LLMs cannot predict response structure (fields, types, pagination). Forces agents to guess at downstream field names.
Descriptions lack WHEN/WHY context. 'Search for papers on arXiv by query, author, category, etc.' does not explain when to use search_papers vs get_paper_details, or what the LLM should do with results.
Error handling returns bare exception text ('Error: {str(e)}') with no recovery guidance. LLM cannot determine if error is retryable, user-fixable, or fatal.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | <=2025-11-25 | v2 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All tools are read-only but this is not declared in schema, forcing LLM to infer safety.
Parameter descriptions lack format/constraint guidance. 'arxiv_id' should specify format (e.g., '5-7 digits, optionally with version suffix like 2301.07041v2'). 'max_results' lacks range (1 - 100?).