Loading ToolBench
Search and traverse the Semantic Scholar Academic Graph, request paper recommendations, and inspect downloadable dataset releases.
17 tools with complete schemas and descriptions. Naming follows verb_noun pattern consistently (get_, search_, batch_, list_). Descriptions are adequate (avg ~120 chars) but lack LLM-optimized guidance on WHEN to use each tool vs. similar ones. Parameters are well-typed with defaults, but many lack constraint documentation (e.g., valid field names, format of publication_date_or_year). Output schemas are not explicitly documented, LLMs must infer structure from API docs. Error handling returns generic {"error": str(exc)} without recovery guidance. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only operations.
Suggest paper-title query completions for a partial query.
Get multiple authors by Semantic Scholar author ID (maximum 1,000).
Get multiple papers by S2, CorpusId, DOI, arXiv, ACL, MAG, PMID, or URL ID.
Run high-volume paper search; pass the returned token for the next page.
Get one author by Semantic Scholar author ID.
List an author's papers, optionally filtering by publication date/year.
Output schemas not documented. LLMs cannot infer what fields to expect from responses (e.g., does search_papers return 'total', 'next_cursor', or 'has_more'?). This forces agents to guess field names for downstream operations.
Parameter constraints not documented. 'fields' parameter accepts arbitrary strings; valid field names are not enumerated or described. 'publication_date_or_year' format is ambiguous (YYYY? YYYY-MM-DD? range syntax?). LLMs will hallucinate invalid values.
Error handling is generic. All errors return {"error": str(exc)} with no recovery guidance. LLMs cannot distinguish retryable (rate limit) from fatal (invalid ID) errors, and have no hint on what to try next.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | <=2025-11-25 | v2 |
Get details about a specific Semantic Scholar dataset release.
List the authors of a paper with offset pagination.
List papers citing a paper, including contexts, intents, and influence.
Get one paper by S2, CorpusId, DOI, arXiv, ACL, MAG, PMID, or URL ID.
List a paper's references, including contexts, intents, and influence.
Get paper recommendations based on a set of paper IDs.
List available Semantic Scholar dataset releases.
Find the paper whose title best matches a supplied title.
Search authors by name, with offset pagination.
Run relevance-ranked paper search with filters and offset pagination.
Search for paper snippets (passages) matching a query.
No tool annotations. All 17 tools are read-only and idempotent, but lack readOnlyHint and idempotentHint annotations. Agents cannot infer safety properties and may over-retry or apply unnecessary confirmation logic.
Descriptions lack LLM-optimized guidance. No explanation of WHEN to use search_papers vs. bulk_search_papers vs. match_paper. No hints on pagination strategy or result limits. Descriptions are factual but not agent-optimized.