Loading ToolBench
MCP server that retrieves academic papers by title — metadata, PDF, full text, citations, references.
Paper Search MCP has well-structured tool definitions with consistent naming (all verb_noun pattern) and comprehensive parameter schemas. However, critical gaps exist: (1) Output schemas are completely undocumented, no response format specifications visible for any of the 13 tools. (2) Tool descriptions vary widely in quality; several are vague about when to use each tool vs alternatives. (3) Error handling is not visible in the provided code, no guidance on what to do if a lookup fails or a PDF is unavailable. (4) Parameter descriptions are present but inconsistent in depth. The server demonstrates solid foundational quality but lacks the production-grade polish needed for confident agent deployment.
Retrieve metadata for multiple papers in a single request using Semantic Scholar paper IDs. Efficient for bulk lookups.
Retrieve comprehensive author profile including name, affiliation, paper count, and h-index using Semantic Scholar author ID.
Retrieve all papers by a specific author. Returns a list of papers authored or co-authored by the given person.
Retrieve the abstract and core metadata for a paper by title. Lightweight operation returning title, authors, year, abstract, and open access status.
Retrieve academic paper metadata and full text by arXiv ID. Fetches metadata from arXiv API and attempts to extract full text from the PDF.
Retrieve academic paper metadata by Digital Object Identifier (DOI). Queries Semantic Scholar and Crossref for comprehensive metadata.
No documented output schemas for any tool. LLMs cannot determine what fields to expect in responses, forcing them to infer structure and potentially miss critical data for chaining calls.
No error handling or recovery guidance visible. Tools that fail (e.g. PDF not found, author lookup returns empty, API rate limit exceeded) provide no actionable guidance to the LLM.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
Retrieve citing papers for a given academic paper by title. Shows the papers that cite this work.
Extract and return the full text content of an academic paper by title. Supports arXiv preprints, PDFs via Semantic Scholar, and open-access sources.
Retrieve comprehensive metadata for an academic paper by title including abstract, authors, citations, references, and open access information.
Fetch the full PDF of an academic paper by title. Returns PDF content, size, number of pages, and extraction status.
Retrieve the references (bibliography) of an academic paper by title. Shows the papers cited by this work.
Find papers similar to a given paper by title. Returns semantically similar papers based on Semantic Scholar's recommendation engine.
Search for academic papers across multiple sources using advanced filters.
Tool descriptions lack 'when to use this vs similar tool' clarity. get_paper_metadata, get_paper_abstract, and get_paper_fulltext all retrieve paper data but their distinctions are not explicitly stated, forcing LLM to reason about functional differences.
No pagination or limit enforcement visible for list-returning tools. search_paper accepts limit (max 100), but it's unclear if get_paper_citations, get_paper_references, and get_author_papers enforce reasonable defaults or caps to prevent context explosion.
Parameter descriptions inconsistently specify constraints and expected formats. For example, arxiv_id description shows format (1706.03762 or 1706.03762v1) but minLength/maxLength constraints may not enforce both variants; DOI description is sparse.