Loading ToolBench
MCP server for arXiv paper search and research management
This server has basic tool definitions with descriptions and schemas, but falls short of production quality in multiple areas. Both tools have clear names starting with action verbs (search_, extract_), and input schemas are present with typed parameters. However, descriptions lack depth on usage context and error scenarios. Parameter descriptions are minimal (under 72 chars baseline). Output schemas are not formally documented, extract_info returns a JSON string without specifying the structure; search_papers returns a list of strings without describing what those strings represent or how to chain them. Error handling is present but returns bare error messages without recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite search_papers being a WRITE operation. The server averages 58/100 across its two tools.
Search for information about a specific paper across all topic directories.
Search for papers on arXiv based on a topic and store their information.
Output schemas not formally documented. extract_info returns a JSON string but the structure (fields, types) is not declared. search_papers returns list[str] but does not specify what each string represents or whether it is a paper ID, title, or formatted entry.
Missing tool annotations (readOnlyHint, destructiveHint) despite clear operational semantics. search_papers is marked Risk: WRITE but has no destructiveHint annotation in the code. extract_info is READ_ONLY but has no readOnlyHint.
Parameter descriptions are minimal and do not meet the 72-character baseline or include usage context. 'The topic to search for' (24 chars) and 'The ID of the paper to look for' (31 chars) lack format constraints, examples of valid values, or dependencies (e.g., paper_id comes from search_papers).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 40 | - | v1 |
Error messages do not provide recovery guidance. 'Error: arXiv API is currently unavailable. Please try again later.' and 'No saved information found for paper' do not tell the LLM what to do next or whether to retry, ask the user, or select an alternative tool.
extract_info has an undocumented dependency on search_papers. The description does not state that paper_id must come from a prior search or that no saved data means search_papers must be called first. This invites misuse.
search_papers returns list[str] without specifying the structure. Are items paper IDs, titles, or formatted summaries? LLM cannot plan downstream calls (e.g., to extract_info) without knowing the format.