Loading ToolBench
Vimmary is the user's personal library of YouTube videos and podcast episodes with AI-generated summaries. It contains summaries and transcripts of YouTube videos the user has watched, bookmarked, or saved, and of podcast episodes from feeds they subscribe to.
Vimmary has 5 tools with complete input schemas and descriptions. Naming follows verb_noun convention (search_videos, get_video, list_recent, stats, resummarize). Descriptions are adequate (100-200 chars) and explain WHEN to use each tool. However, parameter descriptions lack detail on constraints, formats, and valid ranges. The resummarize tool accepts 'level' and 'provider' parameters with no enum constraints or validation guidance. Output schemas are not documented, LLMs cannot infer what fields to expect from responses. Error handling is absent from tool definitions. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles (resummarize is WRITE, others are READ_ONLY).
Get the full summary, transcript, and metadata of a specific video from the user's library. Use after finding a video via search or list to get complete details.
Browse the user's recently watched or bookmarked YouTube videos. Use for 'what did I watch recently', 'latest videos', 'show me my videos'. Supports filtering by channel, language, or topic.
Regenerate the summary for a video with a different detail level.
Search the user's library of YouTube videos and podcast episodes by topic, keyword, or meaning. Use for questions like 'videos about Docker', 'that talk about productivity', 'which podcast covered AI regulation'.
Get statistics about the user's library: total count, breakdown by source, top channels and shows, top topics, and daily activity. Use for 'how many videos do I have', 'what channels do I watch most', 'my top topics'.
Output schemas not documented. LLMs cannot infer what fields get_video, search_videos, list_recent, and stats return. This forces agents to guess at response structure and risks failed downstream tool calls.
resummarize parameter 'level' and 'provider' lack enum constraints and validation guidance. Description says 'medium' or 'deep' but does not enforce it. LLMs may pass invalid values like 'short' or 'verbose'.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint). resummarize is marked WRITE but lacks destructiveHint annotation. Agents cannot determine which tools are safe to retry.
Parameter 'limit' in search_videos and list_recent lacks min/max bounds. No guidance on valid range (e.g. 1-100). LLMs may pass absurd values like 10000.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
No error handling guidance in tool definitions. If search_videos returns no results or get_video fails with 'video not found', LLMs have no recovery path documented.