Loading ToolBench
An MCP server for searching and managing academic papers from arXiv with Claude integration
This server exhibits significant gaps in tool definition quality. While all three tools have basic descriptions and input schemas are present, the descriptions are under-optimized for LLM decision-making, parameter descriptions are minimal or absent, and output schemas are not documented. The tool naming follows verb-first conventions (search_, extract_, generate_) which is good, but parameter typing and constraint documentation are weak. No error handling guidance is provided. The server relies on STDIO transport, which further limits its production viability.
Search for information about a specific paper across all topic directories.
Generate a prompt for Claude to find and discuss academic papers on a specific topic.
Search for papers on arXiv based on a topic and store their information.
Parameter constraints missing: max_results and num_papers have no minimum/maximum bounds declared. Unbounded integers allow LLMs to pass absurd values (0, 10000) that break API calls or cause timeouts.
Output schemas not documented. None of the three tools have formal output schema declarations visible in the code or docstrings. LLMs cannot plan downstream tool calls or extract the right fields when the shape of the response is opaque.
Inconsistent and undocumented error behavior. extract_info returns either a JSON string or a plain text error message depending on whether the paper is found. No error classification (retryable, user-fixable, fatal) and no recovery guidance. LLMs cannot determine what to do next if a call fails.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 43 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 39 | - | v1 |
Descriptions are too short and lack context. search_papers (84 chars), extract_info (102 chars), and generate_search_prompt (71 chars) do not explain WHEN to use each tool, what prerequisites exist, or how they interact. Descriptions should be 50 - 200 chars with explicit context.
State-modifying operations not declared. search_papers writes to disk ('papers/' directory) but the description does not say it modifies state or that calls have side effects. Agents need to know which tools are safe to retry.
Parameter identifiers are opaque. paper_id is passed by the LLM but the format is never explained (arXiv short ID format like '2401.12345'). Agents cannot know how to obtain or format this value without trial-and-error.
No pagination or result limits documented. search_papers respects max_results parameter but returns a list of paper IDs with no mention of a total count or whether more results are available. If an agent searches for a popular topic, it may not realize there are more papers to fetch.