Loading ToolBench
ScholarMCP presents 5 scholarly research tools with generally good naming conventions and documented schemas. All tools start with action verbs (search_, ingest_, get_, extract_, suggest_) following the 90% A+ baseline. Tool descriptions are present and substantive (ranging 180-280 chars), meeting the 10-1024 char guideline and exceeding the baseline average of 194 chars. However, there are notable gaps in parameter documentation consistency. Some parameters lack descriptions or have minimal ones (e.g., 'sections' in extract_granular_paper_details has no description text visible). Output schemas are not explicitly documented in the provided source, forcing the rubric to score conservatively. Error handling guidance is absent from tool descriptions, no indication of what failures look like or how agents should recover. Security considerations are partially addressed (no credentials exposed in parameters), but missing are permission gates, audit trail declarations, and scope declarations. The tools exhibit good composition patterns (each has singular responsibility) and some chaining capability (document_id references), but cross-tool field naming consistency cannot be fully verified without seeing actual responses.
Extract claims, methods, limitations, datasets, metrics, and section-aware summaries from a parsed document.
Get the status of a previously started ingest_paper_fulltext job.
Resolve and ingest a full-text PDF from DOI/URL/local file, then parse into a structured document using GROBID/simple fallback pipeline.
Search multiple scholarly metadata providers (OpenAlex, Crossref, Semantic Scholar, optional Scholar scrape) and return canonicalized paper records.
Recommend citations from the federated literature graph based on manuscript context.
Output schemas not documented. Tool descriptions state WHAT the tool does but do not document return types, field names, or structure. LLMs cannot reliably plan downstream tool calls or extract specific fields without knowing the response schema.
Parameter descriptions incomplete or missing. 'sections' in extract_granular_paper_details has no visible description. 'cursor_context' in suggest_contextual_citations lacks detail on what this field should contain or how it differs from 'manuscript_text'. Parameters named generically ('style', 'k', 'limit') need constraint hints (e.g., 'k must be 1-100').
No error handling guidance in tool descriptions. Tools do not document failure modes (e.g., 'PDF parsing fails if GROBID is unavailable; fallback to simple mode') or recovery steps. Agents have no guidance on whether to retry, adjust parameters, or abandon the path.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 70 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Scope and permission declarations missing. Tools do not specify what permissions or API keys they require (read:arxiv, write:cache, etc.). No audit trail or logging declaration visible in tool definitions, limiting compliance and debugging capability.
ingest_paper_fulltext lacks confirmation/dry-run pattern. This is a stateful, write-heavy operation that modifies the document store. No mention of preview mode or confirmation step before committing. Agents can inadvertently ingest incorrect PDFs.
Result limits not enforced or documented. search_literature_graph defaults to limit=10 (acceptable), but no maximum is stated. suggest_contextual_citations defaults k=10 (good), but no guidance on whether returning 100+ results is possible or would degrade LLM reasoning.