Loading ToolBench
An MCP server for searching and managing academic papers from arXiv with research organization capabilities
Two tools with basic FastMCP definitions. Both have schema (input parameters typed) and descriptions present, but descriptions are generic and lack specificity about WHEN to use each tool, WHAT data is returned, and HOW they interact. No output schemas documented. No error recovery guidance. Parameter descriptions are present but minimal. Naming is reasonable (verb_noun pattern) but tool roles are not clearly differentiated. The search_papers tool has a side effect (writes to disk) that is not highlighted in the description, violating the command-tool pattern.
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a topic and store their information.
Missing output schema documentation. Neither tool documents what fields are returned or in what structure. search_papers returns List[str] (paper IDs) but the description says 'store their information' without clarifying what is stored locally vs returned to the LLM. extract_info returns a JSON string or error message without specifying the JSON structure. LLMs cannot plan chained calls without knowing the output shape.
search_papers description omits side effect disclosure. The tool creates directories and writes JSON files to disk, a destructive/stateful operation, but the description makes no mention of this. Description says 'store their information' vaguely, an LLM cannot tell if this means store locally or return in the response. Per pattern:command-tool, tools that modify state MUST declare this explicitly so agents know they are not idempotent.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 47 | - | v1 |
No error recovery guidance. extract_info returns 'There's no saved information related to paper {paper_id}' on miss, but does not suggest what the user should do next (e.g., 'Call search_papers first to retrieve and cache papers'). search_papers has no documented failure modes or retry guidance. Per pattern:recovery-guide, error messages must tell the LLM what to do next.
Parameter descriptions are minimal and lack format/constraint details. search_papers 'topic' param description is just 'The topic to search for', no guidance on format, length, or constraints. 'max_results' has a default (5) but no documented min/max bounds or rationale. extract_info 'paper_id' has no format description (e.g., is it 'YYMM.NNNNN', or any string?). Per pattern:constrained-input, parameters must describe expected format and allowed values.
Tool descriptions (45 - 48 chars) are below the production baseline of 194 chars average and lack WHEN/WHY context. search_papers: 'Search for papers on arXiv based on a topic and store their information.' does not explain when to call it vs extract_info, or what happens when papers are already cached. extract_info: 'Search for information about a specific paper across all topic directories.' does not explain when the paper must first be indexed by search_papers. Descriptions should be 50 - 200 chars with clear intent signals.
No input validation or constraint enforcement. search_papers accepts any string as 'topic' and any integer as 'max_results', if an LLM passes max_results=10000, arXiv API may reject it or time out, and there is no documented limit or fallback. extract_info accepts any string as 'paper_id' without validating format. Per pattern:tool-gateway, tools must validate inputs and return actionable error messages.
Resource content is not optimized for LLM consumption. get_topic_papers() returns markdown with full paper summaries truncated to 500 chars each. This works but is verbose and token-inefficient. Per mxe:strip-api-responses, responses should strip irrelevant fields and enforce result limits. Consider returning a structured list of papers with title, ID, and one-line summary instead of formatted markdown.