Loading ToolBench
A paper-to-video agent workflow that converts academic papers (arXiv, blogs, PDFs) into animated video presentations with multi-platform distribution (Bilibili, Xiaohongshu, Douyin)
Paperify exhibits severe definition quality gaps across all dimensions. While 14 tools are registered with names and basic descriptions, most lack proper input schemas, parameter descriptions, and output documentation. Tool names are inconsistent (mix of verb_noun and descriptive phrases). Descriptions range from adequate (generate_daily_arxiv_summary: 95 chars) to vague (fetch_creator_stats: 67 chars). Critical issues: 8+ tools have incomplete or missing parameter descriptions; no tools document output schemas; no error handling guidance; no security considerations for API keys (DashScope, Bilibili credentials). The codebase shows tool definitions scattered across multiple files (main.py, paper_discovery.py, distribution/orchestrator.py, etc.) with inconsistent schema rigor. Parameter types are declared but many lack constraints (enums, ranges, format specs). No evidence of idempotency, pagination, or composition patterns.
Fetch all videos from Bilibili channel and merge into published papers cache
Scan output directory for MP4 files and backfill published papers cache with metadata
Auto-pick a paper for a given topic via opencode + heuristics from multiple sources (HuggingFace, arXiv)
Use Qwen-VL (DashScope) to explain and analyze images from papers
Fetch arXiv paper metadata by paper ID
Fetch and extract metadata, images, and videos from blog/article URLs
Pull creator statistics from Bilibili, Xiaohongshu, and Douyin (task #4)
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., what does discover_top_paper return? paper_id? title? authors? link?). Violates pattern:tool and pattern:response-shaper.
Credentials (DashScope API key, Bilibili UID, Douyin/Xiaohongshu tokens) are not documented as server-side injected. If these are passed as parameters, they will leak into agent logs and traces. Violates pattern:secret-injection.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | <=2025-11-25 | v2 |
Generate AI illustration using DashScope Wanx model for intro segment
Generate video summary from arXiv paper with configurable length, language, and target duration
Generate Manim animation presentation from paper with optional TTS narration
Parse arXiv paper ID from arXiv URL
Reply to social media comments via LLM (task #3)
Extract and analyze figures from arXiv LaTeX source code with TikZ/pgfplots support
Upload generated video to multiple social platforms (Bilibili, Xiaohongshu, Douyin)
Parameter descriptions are incomplete or missing for 8+ tools. E.g., explain_image has only 1 param with description; backfill_published_papers has 1 param with minimal context. LLMs cannot infer parameter meaning from names alone. Violates pattern:tool-description.
No error handling guidance. Tools like upload_generated_content (WRITE risk) and reply_to_comments (WRITE risk) do not document what happens on failure, whether errors are retryable, or how to recover. Violates pattern:recovery-guide and pattern:error-classification.
Tool naming inconsistency: 'try_structured_figure' uses 'try' (non-standard); 'backfill_*' uses domain-specific jargon; 'reply_to_comments' is clear but 'fetch_creator_stats' is vague (fetch from where? which platforms?). Violates pattern:tool naming conventions.