Loading ToolBench
MCP server for searching and managing academic papers from arXiv
The server has 5 tools with varying quality. All tools have descriptions and input schemas, but most descriptions are incomplete, ambiguous, or lack critical operational context. Parameter descriptions are sparse or missing. Output schemas are not formally documented. Tool names follow verb-noun conventions (search_papers, extract_infos, get_available_folders, get_topic_papers, generate_search_prompt), but several lack clarity about their actual effects and dependencies. The most critical issue: search_papers has a WRITE risk (modifies filesystem) but its description does not state this consequence, violating the command-tool pattern. extract_infos and other tools do not clearly document their return types or structure. No error handling guidance is present in any tool description.
Search for information about a specific paper across all topic directories.
Generate a prompt for Claude to find and discuss academic papers on a specific topic.
List all available topic folders in the papers directory. This resource provides a simple list of all available topic folders.
Get detailed information about papers on a specific topic.
Search for papers on arXiv based on a topic and store their information.
search_papers has WRITE risk (creates filesystem directories and modifies JSON files) but description does not state this consequence. LLMs cannot determine if this tool is safe to retry or has irreversible side effects.
No tool description explains the return type or structure. search_papers returns 'List[str]' (paper IDs), extract_infos returns JSON string, get_available_folders returns markdown. LLMs need explicit documentation of what fields/structure to expect so they can extract data for downstream calls.
Parameter descriptions are missing or trivial. 'topic' appears in 4 tools but is described identically ('The topic to search for' or 'The research topic to retrieve papers for'). max_results, num_papers lack format/range guidance. paper_id has no hint about format or where to find one.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 51 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
extract_infos returns a bare string ('There's no saved information related to paper...' or JSON). No guidance on error recovery. If paper not found, LLM cannot determine whether to retry, search first, or ask the user.
generate_search_prompt is marked as a 'prompt' (in the @mcp.prompt() decorator) but categorized as a tool in the submission. Unclear if this is a tool or a resource. If it is a prompt template provider, it should not have tool-style input parameters.
get_available_folders and get_topic_papers are resources (not tools), but their descriptions do not explain when to call them or what they enable. 'This resource provides a simple list' is too generic and does not guide LLM selection.