Loading ToolBench
Search arXiv, download papers, and read them back whole or by section. Paper ids are bare arXiv ids such as 2103.12345 or hep-th/9901001.
Three tools with complete schemas and descriptions. watch_topic and check_alerts have strong, actionable descriptions (120-180 chars) explaining the alert mechanism and query syntax. citation_graph is clear but brief (110 chars). All schemas are properly typed with required fields and descriptions. However, parameter descriptions lack constraint details (e.g., arXiv ID format, max_results bounds), and output schemas are not documented. Error handling and recovery guidance are absent from descriptions. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles.
Run the saved topic watches and return the papers published since each was last checked, with full metadata and a count per topic. Each watch's timestamp moves forward afterwards, so the next call only sees newer work. Pass a topic to run one watch, omit it to run them all. Register topics with watch_topic first.
Look up a paper in Semantic Scholar's citation graph and return both directions: the papers citing it and the papers it cites, each with title, year, authors and identifiers. Takes an arXiv ID, and only finds papers Semantic Scholar has indexed.
Save a standing query. check_alerts re-runs it and reports only what has been published since the previous run, so a watch works as an alert for new work on a topic. The query uses search_papers syntax: quoted phrases, ti:/au:/abs: prefixes, AND/OR/ANDNOT. Saving the same topic string again updates that watch instead of adding a second one.
Output schemas not documented. LLMs cannot plan downstream calls or extract fields without knowing what citation_graph, check_alerts, and watch_topic return. Responses are opaque.
Parameter constraints missing. 'paper_id' lacks format spec (arXiv ID syntax). 'max_results' has default=10 but no min/max bounds. 'topic' query syntax is explained but not validated. LLMs will pass invalid values.
No error recovery guidance. Descriptions do not explain what happens on invalid arXiv ID, network failure, or Semantic Scholar indexing gaps. LLMs have no recovery path.
Tool annotations missing. watch_topic is WRITE (destructive), check_alerts is READ_ONLY, citation_graph is READ_ONLY. Annotations not declared in schema, forcing LLMs to infer from description.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | <=2025-11-25 | v2 |
No pagination or result limits documented. check_alerts may return unbounded results across all watches. citation_graph may return hundreds of citing/cited papers. Context window risk.