Loading ToolBench
CLI and MCP server for Substack publisher metrics (subscribers, opens, CTR, revenue, audience overlap)
Server has 14 well-intentioned tools with consistent naming (verb_noun pattern) and reasonable descriptions (avg ~150 chars). All tools are READ_ONLY, reducing risk. However, critical gaps exist: NO input schemas are visible in the source code, parameter types and descriptions are inferred from function signatures only, not declared in tool registration. Output schemas are completely undocumented. Error handling is absent, no guidance on what to do if Substack API fails or auth is invalid. Parameter descriptions lack constraints (e.g., 'days' has no min/max bounds). The 'full_report' tool combines multiple concerns. Pagination is present but not fully documented in all tools.
Audience insights: follower time series, US state breakdown, country totals, and publications whose readers overlap with yours (for collaboration/cross-promo targeting).
Publications whose readers overlap most with yours. High-signal for collab targeting.
Daily email delivery counts.
Kitchen-sink report calling every metric endpoint in one pass. Large response — use when you want a complete snapshot for analysis or archiving.
Subscriber acquisition: signup sources ranked by user count, milestone events, and daily growth time series.
Percentage of subscribers acquired via Substack's internal network (recommendations, app, discover feed).
NO input schemas visible in source code. Tools use FastMCP decorators with Python type hints, but no explicit JSON Schema registration is shown. This prevents LLMs from understanding parameter constraints, types, and defaults at the protocol level.
Output schemas completely undocumented. Tools return dicts/lists with no declared structure. LLMs cannot plan downstream calls or extract fields reliably. E.g., 'summary()' returns a dict, what fields? 'posts()' returns a dict, does it include pagination metadata?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 42 | 2026-07-28+ | v2 |
Rolling 30-day open rate across all posts.
Pending paid-subscription pledges (readers who've committed to pay once you enable paid plans).
Counts of published, draft, and scheduled posts.
Per-post email analytics: opens, open rate, clicks, click-through rate, 7/28-day unique opens, engagement rate, estimated value, signups driven, unsubscribes, likes, comments, restacks, views, and more. Up to 20 posts per call — paginate with offset.
Publication metadata: name, id, active plans and prices, payment settings.
Full subscriber list with email, name, subscription type (free/monthly/annual/lifetime/founding/gift/comp), activity rating (1-5 stars), revenue generated, and join date.
Top-line publisher stats: subscriber counts, paid subs, ARR, and total views at the start and end of the window. Good first call to understand a publication's current state.
Traffic summary: 30-day headline number, daily time series, and breakdown by referrer source (direct, social platforms, email, Substack app, search, etc.).
Parameter constraints missing. 'days' parameter appears in 7 tools with no min/max bounds, LLMs could pass 0, -1, or 999999. 'limit' and 'offset' similarly unbounded. No enum constraints on any parameter.
No error handling or recovery guidance. If Substack API fails, auth is invalid, or rate limits are hit, tools return raw exceptions. LLMs receive no actionable guidance on retry, fallback, or user-fixable steps.
'full_report' combines 14 separate concerns into one tool. Violates single-responsibility principle. If one metric fails, the entire report fails. LLM cannot compose individual metrics as needed.