Loading ToolBench
Model Context Protocol server for Semantic Scholar API
Scholar Search MCP presents a mixed profile. Naming is strong and verb-driven (search_papers, get_paper, arxiv_download_source). Descriptions are present for all 9 tools and generally adequate (80-150 chars), meeting the baseline. However, input schemas visible in the submission are incomplete: parameter types are declared (string, integer, array) but many lack depth in constraints, ranges, or field-level descriptions. The arxiv_download_source tool accepts a 'output_dir' parameter with no validation constraints documented; the tool performs file I/O with path normalization logic, but the schema does not expose these constraints to the LLM. Parameters like 'limit' and 'offset' across multiple tools lack explicit min/max bounds, relying on inline comments in source (e.g., 'max 100') rather than formal JSON Schema constraints. Output schemas are not documented, LLMs cannot reason about what fields are returned or how to chain results to downstream tools. Error handling is implementation-aware (e.g., tarfile.TarError, 404 from arXiv) but error messages in the code are descriptive; however, there is no evidence of error recovery guidance in the tool definitions. The code implements caching and retry logic (MAX_429_RETRIES=6), but this is not exposed as a tool feature, agents cannot reason about idempotency or retry-safety. Overall, this server demonstrates competent naming and basic description coverage, but falls short of production-grade schema completeness and error-handling clarity that would enable robust LLM composition.
Download and extract LaTeX/source bundle from arXiv
Search for papers on arXiv
Get detailed information about a specific author
Get papers authored by a specific author
Get detailed information about a specific paper
Get citations of a specific paper
Get references of a specific paper
No output schemas documented for any tool. LLMs cannot reason about returned fields, pagination structure, or how to chain results to downstream tools. For example, search_papers likely returns an array of papers with fields like 'paperId', 'title', 'citationCount', but this is invisible to the agent, forcing it to guess at field names and miss opportunities for composition.
Numeric parameters (limit, offset, start, max_results) lack formal min/max constraints in the schema. Limits are mentioned in descriptions ('default 10, max 100') but not as JSON Schema constraints (minimum, maximum). LLMs cannot validate their own inputs against these bounds and may pass invalid values.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 48 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 23 | - | v1 |
Search for authors in Semantic Scholar
Search for papers in Semantic Scholar
The 'fields' parameter (array type) has no description of valid field names, expected structure, or defaults. A user/agent cannot determine what values to pass without consulting external Semantic Scholar documentation. This violates the pattern that parameters should include dependency hints and valid value guidance.
arxiv_download_source performs filesystem operations (extraction, path normalization) but the output_dir parameter lacks validation constraints or security guidance. The description does not warn about path traversal risks, symlink attacks, or disk space limits. The code implements safeguards (_safe_extract_tar_gz, path validation), but these are not exposed as tool-level guarantees.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) declared. The protocol supports these; 8 of 9 tools are read-only and should be annotated as such. arxiv_download_source is destructive (writes to disk, removes existing directories) and should be marked destructive. This prevents agents from reasoning about tool safety and retry-ability.
Error handling is implementation-aware but not agent-aware. The code raises ValueError with messages like 'No source package at {url} (404)' and 'Unsafe path in archive', but there is no guidance for LLMs on recovery actions (e.g., 'Try searching for an alternative paper' or 'Retry with different output_dir'). Errors do not classify themselves as retryable, user-fixable, or fatal.
No pagination structure documented. Tools accepting limit/offset (search_papers, get_paper_citations, etc.) should document whether they return a 'total' count or 'has_more' flag. Without this, agents cannot reason about whether more results exist or how to fetch the next page safely.