Loading ToolBench
FastAPI application for generating podcast-style scripts and audio from academic papers
PodcastAI exposes 4 tools via FastAPI with mixed quality. All tools have descriptions and basic input schemas, but multiple critical issues prevent this from being production-grade: (1) Tool names lack action verbs, 'get_task' is weak, 'request_paper_generation' is awkwardly long; (2) Parameter descriptions are minimal (10-30 chars) and lack actionable constraints; (3) No output schemas are documented, LLMs cannot plan downstream calls; (4) No error handling guidance or recovery paths; (5) Security concerns: authentication tokens passed as implicit authorization without scoping; (6) Tool composition issues: signup/login return raw JWTs without user context; (7) No pagination or result limits documented despite streaming responses.
Check generation status and retrieve task details
Authenticate a user and return an access token
Submit a paper link for processing. This endpoint will accept a paper link and initiate the generation process.
Register a new user and return an access token
NO OUTPUT SCHEMAS DOCUMENTED. request_paper_generation returns StreamingResponse, get_task returns untyped dict, signup/login return Token but with minimal fields. LLMs cannot plan chaining calls, extract IDs, or validate results.
PARAMETER DESCRIPTIONS ARE TOO BRIEF (10-30 chars). Lack actionable constraints, format requirements, and error cases. E.g., 'link' parameter for request_paper_generation has no URL validation guidance, no PDF format confirmation, no size limits.
WEAK TOOL NAMING. 'request_paper_generation' is verbose and awkwardly phrased; 'signup' lacks resource context; 'get_task' is generic. Should follow verb_noun pattern: 'submit_paper', 'create_user', 'fetch_task_status'.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 44 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 38 | - | v1 |
NO ERROR HANDLING GUIDANCE. Tools lack descriptions of failure modes, retryability, or recovery paths. E.g., what does the LLM do if paper link is invalid, task_id expires, or authentication fails?
SECURITY: Credentials (tokens) returned in responses and potentially logged. Token includes exp datetime but no scopes or permissions. No audit trail of who authenticated or when.
MISSING RESPONSE IDs FOR CHAINING. signup/login return Token but no user_id, email confirmation, or account status. get_task returns task details but unclear what IDs/URLs are available for follow-up calls (e.g., how to retrieve generated podcast audio).
STREAMING RESPONSE UNDEFINED. request_paper_generation returns StreamingResponse but tool description does not explain audio format, bitrate, or whether it's a live stream vs immediate file download. LLM cannot plan how to deliver to user.
NO PAGINATION OR RESULT LIMITS. get_task returns dict(tasks) but no limit or offset params. If 'tasks' is a collection, it could be unbounded and blow context window.
TOOL COMPOSITION INCOMPLETE. No tool exists to retrieve generated podcast script, audio, or metadata separately. get_task returns 'task details' but unclear what that includes or how to extract artifacts.