Loading ToolBench
An MCP server for searching and extracting academic papers from arXiv, with support for organizing papers by topic and generating research prompts.
This server has two tools with basic schemas and minimal descriptions. Both tools have explicit JSON Schema definitions in tool_schema.py, but descriptions are under-detailed for LLM optimization. Parameter descriptions are present but sparse. No output schema documentation. No error handling guidance. The implementation in L4/tool.py uses FastMCP decorators but actual tool registration is incomplete (resource registration is cut off mid-definition). Critical issue: descriptions lack context about when to use each tool, what they return, and prerequisites.
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a topic and store their information.
Tool descriptions are under 50 characters and lack LLM-optimization guidance. 'Search for papers on arXiv based on a topic and store their information' (71 chars) does not explain WHEN to call this vs extract_info, what the return value structure is, or what happens to state.
No output schema documentation. search_papers returns List[str] (paper IDs), but the LLM does not know the structure, format, or how to use paper IDs in extract_info. extract_info returns either a JSON string or an error message, this inconsistency is not documented. Per pattern:tool, LLMs need to know what fields to expect.
Parameter descriptions are minimal. 'topic' is described as 'The topic to search for', this does not explain format, examples, or constraints. 'max_results' has a default of 5 but no explanation of why, or valid range (is 1000 allowed? Does it degrade performance?). 'paper_id' has no format guidance, is it 'arxiv:2401.00001' or '2401.00001' or '2401.00001v2'?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 45 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 46 | - | v1 |
No error handling guidance. extract_info returns a plain string 'There's no saved information related to paper {paper_id}.' on failure, this does not tell the LLM what to do next. Should it retry with a different paper_id? Call search_papers first? Per pattern:recovery-guide, errors must guide the LLM to recovery steps.
search_papers modifies state (writes to disk in 'papers' directory) but the description does not clarify this. The tool stores information in topic-specific JSON files, yet the description says nothing about side effects, idempotency, or what happens if the directory already exists. Per pattern:command-tool, state-modifying tools must declare their side effects.
Tool registration in L4/tool.py is incomplete. The resource registration code is cut off mid-definition ('@mcp.resource("papers://folders") def get_a'). Cannot verify if resources are properly declared. This blocks assessment of whether the server fully implements the MCP tool registration contract.
No input validation or constraint documentation. max_results defaults to 5 but has no minimum/maximum bounds stated in the schema or description. An LLM could pass max_results=999999 and cause performance degradation. paper_id has no regex pattern or format constraint.
No structured output definition. extract_info returns 'JSON string with paper information' but the schema of that JSON (keys like title, authors, summary, pdf_url, published) is not documented in the tool output schema. Per pattern:response-shaper, tools returning objects must document the field structure so LLMs know how to extract data.