Loading ToolBench
MCP server for searching and extracting information from arXiv research papers
This MCP server has minimal definition quality. While both tools are explicitly registered with FastMCP decorators and have descriptions, the descriptions are vague and lack clarity about when to use each tool vs alternatives. The schema is visible but sparse: parameters lack detailed constraints, enums are missing where appropriate (e.g., no limit on max_results), and there is no documented output schema. Error handling is minimal, extract_info returns a bare string message instead of structured error guidance. The tool composition creates a problematic dependency: search_papers writes to disk, while extract_info reads from disk, but the interface gives no hint of this coordination. No validation, rate limiting, or security considerations are evident. Parameter descriptions are too brief (topic: 'The topic to search for' is only 25 chars, paper_id: 'The ID of the paper to look for' is 33 chars, both below the 72-char baseline and offer no context on format or constraints).
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a topic and store their information.
No output schemas documented. Both tools return untyped strings or lists, making it impossible for LLMs to plan downstream operations or extract structured fields.
Error handling is non-actionable. extract_info returns a plain string message ('There's no saved information...') with no guidance on next steps. LLMs cannot distinguish success from failure or know whether to retry or ask the user.
Parameter descriptions are too brief and lack constraints. 'The topic to search for' (25 chars) and 'The ID of the paper to look for' (33 chars) offer no format hints, valid ranges, or examples. No guidance on expected paper_id format (arXiv ID like '2301.12345' vs short ID).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 43 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 34 | - | v1 |
Hidden dependency between tools not documented. search_papers writes to disk; extract_info reads from that disk cache. LLMs have no way to know this coupling from the tool definitions alone, risking calls to extract_info before search_papers or with no matching cached papers.
search_papers is a command tool (writes state to disk) but its description does not warn of this. Pattern:command-tool requires explicit state-modification warnings so agents know the call is not idempotent and has side effects.
No input validation or constraints. max_results has no minValue or maxValue; passing 0 or 10000 is technically allowed. topic is a free-form string with no length limits or format hints. This invites invalid inputs and wasteful API calls.
Ambiguous return type for extract_info. Returns either a JSON string (on success) or a plain error message (on failure). LLMs cannot parse this without explicit type hints. The response should be structured (e.g., {status: 'success'|'not_found', data: {...}, error: '...'})
extract_info does not hint at available alternatives when a paper is not found. Should enrich error: 'Paper XXXX not found. Papers available in topic folders: machine_learning (5 papers), nlp (3 papers).'