Loading ToolBench
A server for searching and extracting information from arXiv papers
The server defines 2 tools with basic FastMCP decorators. Both tools have non-empty descriptions and visible input schemas, but the descriptions lack depth and fail to guide LLM decision-making. Tool naming follows verb_noun convention (get_*, extract_*) but descriptions are too brief (<100 chars) to meet production standards. Parameters are typed but lack detailed descriptions explaining format, constraints, or dependencies. Output schemas are undocumented, the LLM must infer return types from code. Error handling is minimal: tools return plain strings rather than structured error objects with recovery guidance. The server also implements resources and prompts (good composition), but these do not elevate tool quality. Overall: functional but well below A-grade standards.
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a given topic.
Tool descriptions are too brief (under 100 chars each) and lack context for LLM selection. 'Search for papers on arXiv based on a given topic' does not explain when to prefer get_arxiv_papers over extract_info, what structure is returned, or what happens after (file I/O side effect).
Output schemas are not documented. Tool descriptions do not specify return types or structure. get_arxiv_papers returns List[str] (paper IDs), but LLMs cannot infer this from the docstring, they must read Python code. extract_info returns a JSON string or error message, mixing structured and unstructured output with no schema declaration.
Parameter descriptions are minimal. 'topic' is described as 'The topic to search for' (4 words, 23 chars), no mention of format constraints, valid length, or examples. 'max_results' lacks guidance on reasonable bounds or default justification. 'paper_id' is described as 'The ID of the paper to look for', no clarification on expected format (arXiv ID pattern like '2301.12345').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 40 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 38 | - | v1 |
Error handling returns plain strings instead of structured error objects. When a paper is not found, extract_info returns 'There\'s no saved information related to paper {paper_id}.', no error code, no suggestions for recovery. LLMs cannot distinguish user error from system error or know whether to retry.
get_arxiv_papers has an undocumented side effect: it creates a directory and writes papers_info.json to disk (WRITE risk). The description does not state this is a destructive operation that modifies state. Agents will call it assuming it is read-only, leading to unexpected file system changes.
No input validation or constraint documentation. 'max_results' defaults to 2 but no bounds (min/max) are declared. An LLM could pass max_results=10000, causing the arxiv client to hang or fail. Parameter descriptions should include range constraints like '1-100'.
No pagination or result limits documented. get_arxiv_papers returns up to max_results paper IDs (default 2, but no stated cap). For large searches, returned lists could exceed context windows. The tool description should state the maximum result size and offer pagination guidance.