Loading ToolBench
A Model Context Protocol server for monitoring MCP server registry changes and sending notifications via multiple channels (email, webhooks, RSS/Atom feeds)
MCP Notify presents a well-structured HTTP service with 21 tools covering subscription management, change monitoring, and feed delivery. All tools have names starting with action verbs (Create, List, Get, Update, Delete, Pause, Resume, Test) which is positive for LLM disambiguation. Descriptions are present for all 21 tools and are substantive (ranging 40-120 chars, within the 10-1024 baseline). However, there are critical gaps in parameter schema completeness and output schema documentation. Most tools declare input parameters with types and descriptions, but the output schemas are not visible in the source code provided, only inferred from handler filenames. The subscription and change management tools are well-designed for composition (CreateSubscription → ListSubscriptions → GetSubscription → UpdateSubscription), but the feed tools (RSSFeed, AtomFeed, JSONFeed) have zero input parameters documented, raising questions about how filtering or pagination is controlled. Error handling guidance is not visible in the source snippets. Security aspects (secrets, permission gates, audit trails) cannot be verified from the provided code. The Dockerfile and Makefile show good DevOps practices but do not illuminate tool behavior.
Get changes as an Atom feed for subscription in feed readers
Create a new subscription to monitor MCP servers for changes
Delete a subscription
Get details of a specific detected change
Get details of a specific server from the registry
Get all detected changes for a specific server
Get global statistics about the MCP Notify service
Get details of a specific subscription by ID
Feed tools (RSSFeed, AtomFeed, JSONFeed) have empty input schemas. No parameters documented for filtering, pagination, date range, server selection, or subscription filtering. LLM cannot understand how to customize feed content.
Output schemas not visible in source code. Only handler function names suggest what each tool returns (e.g., internal/api/handlers/subscriptions.go). Cannot verify that responses include all fields needed for tool composition (e.g., does CreateSubscription return subscriptionID for use in UpdateSubscription or DeleteSubscription?).
No visible error handling guidance in provided source code. Cannot verify that error responses tell LLMs what to do next (retry, ask user, use alternative tool, or fail). Pattern-tool requires clear error classification.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 57 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 39 | - | v1 |
Get notification history for a subscription
Check if the service is healthy
Get changes as a JSON feed
List detected changes across all monitored servers
List all servers available in the MCP registry
List all subscriptions with pagination support
Pause a subscription to temporarily stop notifications
Get changes as an RSS feed for subscription in feed readers
Check if the service is ready to accept traffic
Resume a paused subscription
Send a test notification to verify subscription channels are working
Test a webhook endpoint by sending a test payload
Update subscription settings
Subscription filtering mechanism unclear. CreateSubscription accepts a 'filters' parameter of type 'object' with description 'Filters for subscription (servers, change types, etc.)' but no nested schema visible. LLM cannot know what filter fields are valid (e.g., is it {servers: [], changeTypes: []} or {serverNames: [], eventTypes: []}?).
Notification channels parameter is an array of unspecified type. CreateSubscription and UpdateSubscription accept 'channels' of type 'array' with description 'Notification channels for this subscription' but no item schema. Is it an array of strings (channel names), objects (with type and config), or IDs? LLM cannot construct valid requests.
No visibility into permission gates or access control. Tool definitions lack scope declarations (e.g., 'read:subscriptions', 'write:subscriptions', 'delete:subscriptions'). Cannot verify that destructive tools (DeleteSubscription) are gated behind authorization checks.
GetStats and Health/Ready tools have minimal descriptions (60 chars or less) and no visible output schemas. An LLM calling GetStats cannot know what statistics are returned (subscription count, change volume, API latency, uptime percentage, etc.?).
No documentation of pagination limits. ListSubscriptions, ListChanges, ListServers accept 'limit' parameters but descriptions do not specify min/max (e.g., must limit be 1-100? 1-1000? Is there a default?). Unbounded integers let LLMs request absurdly large result sets.
Composition chains not explicitly documented. If CreateSubscription returns a subscriptionID, this must be documented so LLM knows it can pass that ID to UpdateSubscription, DeleteSubscription, TestSubscription, etc. Verify response includes all IDs needed for downstream calls.