Loading ToolBench
MCP server for searching Lenny's Podcast transcripts - 284 episodes of product wisdom
Three tools with consistent naming conventions and reasonable descriptions, but schemas lack full specification of return types. Tool names follow verb_noun pattern (search_, get_, list_), and descriptions provide context for when to use each tool. However, parameter descriptions are sparse, output schemas are entirely undocumented, and there is no error recovery guidance. The server implements basic error handling but does not categorize errors or guide agents toward recovery actions. For a podcast transcript search service, the tool definitions are functional but fall short of production-grade quality, descriptions average ~150 chars (within baseline of 194) but lack specificity about return formats and parameter constraints.
Get the full transcript for a specific episode by guest name. Use this when you want to dive deeper into a specific conversation.
List all available episodes/guests in Lenny's Podcast archive. Use this to see what guests and topics are available to search.
Search across all of Lenny's Podcast transcripts for insights on a topic. Returns relevant excerpts from episodes with guest names. Use this to find what product leaders and experts have said about specific topics like pricing, growth, product management, hiring, etc.
Output schemas are not documented. Tool descriptions do not specify the structure of returned results, forcing LLMs to infer field names from examples. For search_transcripts, the code returns { guest, snippet } but this is not declared in the inputSchema. For get_episode, it returns { guest, content } but this is opaque to the agent.
Parameter 'limit' in search_transcripts lacks type definition and constraint. Schema declares it as 'number' but does not specify min/max bounds. Code defaults to 10 and accepts any number, but there is no limit enforcement visible, risking unbounded or negative values.
Error responses do not guide recovery. When get_episode fails with 'Episode with guest X not found', the response includes suggestions but does not explicitly say 'Call list_episodes() to see available guests' or 'Try search_transcripts() to find mentions of this guest'. Error messages are informative but lack actionable next steps for the agent.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 45 | - | v1 |
Parameter descriptions are minimal. The 'query' parameter in search_transcripts says 'use keywords related to the topic' but does not explain format constraints (min/max length, forbidden chars, special syntax). The 'guest' parameter in get_episode provides an example ('Shreyas Doshi', 'Julie Zhuo') but the description does not clarify whether partial names work, case sensitivity, or exact-match requirement.
list_episodes tool takes no parameters and returns a raw array of guest names as text. This forces the agent to parse unstructured output or call get_episode to explore. A structured response (array of { guest_name, episode_number, date } objects) would enable filtering and discovery without extra calls.
search_transcripts does not document pagination. The 'limit' parameter caps results, but there is no offset, cursor, or indication of total count. If an agent searches for a broad topic and gets 10 results (the default), it cannot determine whether there are 11 results or 1,000. Without pagination info, the tool is difficult to compose into multi-step workflows.