Loading ToolBench
AI-based academic paper search, review, and analysis system with FastAPI backend and React frontend. Provides REST API for paper discovery, deep review analysis, visualization, and blog publishing.
Server has 17 tools with basic schemas and descriptions, but significant gaps prevent higher scores. Most tools have descriptions (10 - 200 chars), but many lack depth on WHEN to use them or prerequisites. Input schemas are present but sparse: most parameters lack type constraints (enums, min/max). Output schemas are not documented anywhere in the source. Error handling is absent, no recovery guidance, no categorization of retryable vs fatal errors. Tool naming is generally verb-first (search, deep_review, extract, publish), but some are vague (visualize_poster, generate_poster_figures). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels. Composition is weak: tools like publish_blog_post and submit_indexnow should be one atomic operation, not two separate calls. No evidence of pagination support for list-returning tools.
Perform advanced paper analysis using LightRAG (Graph RAG) for knowledge extraction and reasoning
Add or remove paper from user's bookmarks with tagging and organization
Build and analyze citation network graph from papers with embedding-based similarity and community detection
Interactive chat interface for asking questions about papers with source-grounded responses
Automatically discover and collect referenced papers from a paper's bibliography
Create a structured learning curriculum from a collection of papers with suggested reading order
Generate AI-powered deep review comparing multiple papers with claim verification against source passages
No output schemas documented. Tools return data but LLMs cannot plan downstream calls without knowing response structure. E.g., search returns papers but fields (paper_id, title, authors, url) are not specified.
Missing tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools are marked with Risk levels (READ_ONLY, WRITE) in metadata but not exposed to LLM via schema annotations. LLM cannot infer safety properties.
No error handling or recovery guidance. Tools lack descriptions of failure modes, retryability, or what the LLM should do if a call fails. E.g., search may fail due to network; publish_blog_post may fail due to invalid slug.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
Export paper data and reviews in multiple formats (JSON, CSV, BibTeX)
Extract full text content from PDF papers using PyMuPDF and pdfplumber
Generate academic diagram figures for poster using paperbanana diagram generation
Retrieve analytics events for admin dashboard with filtering and aggregation
Get personalized paper recommendations based on reading history and preferences
Publish a blog post with markdown content, SEO metadata, and automatic IndexNow submission
Search academic papers across multiple sources (arXiv, Google Scholar, Semantic Scholar) with relevance filtering and ranking
Generate shareable link for a paper review or curriculum with access control
Submit published blog URLs to IndexNow for instant indexing by search engines
Generate academic poster visualization from paper analysis with HTML to PDF/PNG export via Chromium
Parameter constraints missing. 'max_results' (search), 'limit' (get_recommendations), 'expiration_days' (share_review) lack min/max bounds. LLM can pass absurd values (max_results=999999, expiration_days=-1).
Weak tool composition. publish_blog_post and submit_indexnow are separate tools but almost always called together. Should be one atomic operation or submit_indexnow should be called automatically after publish succeeds.