Loading ToolBench
A FastMCP server for searching and extracting academic papers from arXiv
This server has moderate structural quality but significant gaps in parameter and output documentation. All 5 tools are present with descriptions and basic schemas, but lack depth in error handling, output documentation, and parameter constraints. Tools follow a sensible semantic pattern (search/get/extract) and are named with action verbs. However, parameter descriptions are minimal, output schemas are not formally documented, and error recovery guidance is absent. The server demonstrates HTTP transport and basic FastMCP integration but falls short of production-grade standards.
Search for information about a specific paper across all topic directories.
Generate a prompt for Claude to find and discuss academic papers on a specific topic.
List all available topic folders in the papers directory.
Get detailed information about papers on a specific topic.
Search for papers on arXiv based on a topic and store their information.
Output schemas not documented. Tools return strings or lists without formal schema declarations. LLMs cannot infer what fields are present in returned JSON, forcing them to parse unstructured text.
Error handling lacks recovery guidance. extract_info returns a plain string for not-found cases; get_topic_papers returns markdown with generic messages. No indication to the LLM what to try next or how to recover.
Parameter descriptions are sparse. 'topic' appears in 3 tools but is minimally described ('The topic to search for', 'The research topic'). No constraints on length, format, or examples of valid inputs. LLMs lack guidance on valid values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 63 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 51 | - | v1 |
No input validation or constraint declarations. max_results and num_papers parameters have defaults but no min/max bounds. Could a user pass max_results=10000 or num_papers=-1? No validation shown in code.
Tools return markdown or plain text when structured JSON would be clearer. get_available_folders and get_topic_papers return markdown strings; extract_info returns raw JSON wrapped in a string. Inconsistent formats force LLMs to parse multiple output types.
Tool descriptions lack context on when to use them. What distinguishes get_topic_papers from extract_info? When should an LLM call get_available_folders vs search_papers? No guidance on tool selection strategy.
search_papers modifies state (writes to disk) but the description does not explicitly say so. Agents need to know this is not a read-only operation to reason about idempotency and side effects.
generate_search_prompt is labeled as a prompt (mcp.prompt()) but semantically it is a tool that generates text. The naming 'generate_search_prompt' does not clarify that this is a prompt generator vs a search tool. Ambiguous semantic role.