Loading ToolBench
Server has 4 tools with basic definitions, schemas, and descriptions. All tools are properly registered with input schemas and descriptions. However, several patterns are missing or incomplete: (1) Parameter descriptions lack constraint details (min/max, format expectations); (2) Output schemas are not documented, callers don't know what fields to expect from each tool; (3) Error handling is minimal, McpError is used but recovery guidance is absent; (4) No tool annotations (readOnlyHint/idempotentHint) despite all being read-only operations; (5) Tool names use vendor prefix pattern (fireflies_*) which is acceptable but descriptions could be more prescriptive about WHEN to use each tool vs similar ones. All tools follow verb_noun naming (get_, search_, generate_) which is good. Descriptions range 50-84 chars, within the baseline p10=34/p90=392 but on the minimal end of what LLMs need for disambiguation.
Generate a summary of a meeting transcript
Retrieve detailed information about a specific transcript
Retrieve a list of meeting transcripts with optional filtering
Search for transcripts containing specific keywords
Output schemas are not documented. Tool descriptions do not specify what fields callers should expect (e.g., does fireflies_get_transcripts return [{ id, title, date, speakers, duration, url }]?). LLMs cannot plan downstream data extraction or chaining without knowing response structure.
Parameter descriptions lack constraint details. 'limit' has no min/max bounds (e.g., is 0 valid? is 1000 allowed?). 'from_date' and 'to_date' say 'ISO format (YYYY-MM-DD)' but don't specify if null is allowed or what happens if from_date > to_date. 'format' enum is specified but not explained when to use bullet_points vs paragraph. These ambiguities force LLMs to guess.
Tool annotations missing. All four tools are read-only (no side effects, safe to retry), but none declare readOnlyHint: true in their definitions. This forces LLMs to infer safety from description text, which is error-prone. MCP spec supports tool annotations, use them.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 54 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 53 | - | v1 |
Error handling lacks recovery guidance. The code throws McpError with ErrorCode.InvalidParams or ErrorCode.InternalError, but responses don't tell the LLM what to do. Example: if transcript_id doesn't exist, the error should be 'Transcript ID "xyz" not found. Call fireflies_search_transcripts() to find a valid ID.' Currently just 'Resource not found'.
Tool descriptions don't disambiguate when to use each tool. 'fireflies_get_transcripts' vs 'fireflies_search_transcripts', the first filters by date, the second by keywords, but descriptions don't say 'Use this when you need keyword matching' or 'Use this for date range queries'. LLMs will pick arbitrarily.
Search implementation is client-side filtering. Code comment states 'Fireflies doesn't have a direct search endpoint, so we'll fetch transcripts and filter them by title (this is a simplified approach)'. This is incomplete, searching sentences isn't implemented, and fetching 20 transcripts client-side and filtering is inefficient. LLMs will call this expecting full-text search.