Loading ToolBench
MCP server over 1,100+ SaaS/cloud vendor status pages — current state, incident history, resolution times and feed URLs, polled daily. No key, no dependencies.
Seven read-only tools with clear, domain-specific names (find_vendor, vendor_status, broken_now, vendor_incidents, vendor_mttr, vendor_map, vendor_feed_urls). All tools have descriptions (194 - 250 chars, within baseline 34 - 392 range). Input schemas present with type declarations and parameter descriptions. However, output schemas are not documented in the source code, responses are inferred from descriptions only. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only. Parameter descriptions are adequate but lack explicit format constraints (e.g., 'vendor' accepts slug/name/domain but no enum or pattern). Error handling is implicit (fetch() falls back to stale cache) but not surfaced to the LLM.
List every vendor that is not ok right now, ranked by severity (major incidents first). Includes status, incident count, and poll timestamp for each.
Resolve a free-text vendor query to a vendor record: slug, name, status-page URL, platform, and machine-readable feed URLs. Fuzzy-matches on slug, name, domain, and token overlap; exact matches rank first.
Get machine-readable feed URLs for a vendor by platform: JSON endpoints, RSS, Atom, etc. Only conventions verified live are listed; bespoke vendors may have none.
Pull a vendor's incident history: list of incidents with start time, resolution time (if resolved), duration, and title. Sorted newest first.
List all vendors in the map with their metadata: slug, name, status-page URL, platform, and whether machine-readable feeds are supported. Useful for discovery and bulk queries.
Compute how long a vendor's incidents usually take to resolve: median, mean, min, max, and count of resolved incidents. Useful for SLA planning.
Output schemas not documented. Tool descriptions state what is returned (e.g., 'status, incident count, poll timestamp') but no formal JSON Schema for responses. LLMs cannot plan downstream tool calls or extract fields reliably without documented output structure.
No tool annotations despite all tools being read-only and idempotent. Missing readOnlyHint and idempotentHint annotations prevent clients from optimizing caching, retry logic, and safety checks.
'vendor' parameter in 5 tools accepts slug/name/domain but no enum or pattern constraint. Description says 'Vendor slug, name, or domain' but LLMs cannot validate input format without formal constraints. Should document accepted formats explicitly or provide a discovery tool (vendor_map) as prerequisite.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | <=2025-11-25 | v2 |
Read the current state of a vendor from the newest poll: status (ok/major/partial/degraded/maintenance/unknown), when it was polled, and the incident count.
Error handling is implicit (fetch() silently falls back to stale cache on network failure). LLM receives no signal that data is stale or that a network error occurred. Tool descriptions mention 'as_of' and 'source' but no error response structure is documented.
No pagination support. 'vendor_map' returns all 1,100+ vendors in a single response. 'vendor_incidents' accepts a 'limit' parameter (good) but no offset/cursor for pagination. Large result sets risk context window exhaustion.