Loading ToolBench
MCP server exposing SRE knowledge primitives (KB search, incident similarity, scenario library, alert parsing) as tools for incident analysis. Provides data tools, persistence, and utilities for structured incident response without running the LLM on the server side.
Server has 6 tools with clear naming (verb_noun pattern) and reasonable descriptions (100-250 chars). Input schemas are present with type definitions and constraints (min/max for limits). However, output schemas are not documented, LLMs cannot predict return structure. Error handling is minimal; tools lack recovery guidance. Parameter descriptions are adequate but could be more prescriptive about formats and constraints. Missing tool annotations (readOnlyHint/destructiveHint). One tool (save_incident_analysis) accepts a complex nested 'analysis' object but the schema reference to 'AnalysisSchema' is not visible in source, forcing inference.
Search the incident history for past events with similar service, symptoms, or root cause. Returns up to 5 ranked by similarity. Use the title+symptoms+summary of a new incident as the query.
Returns the complete raw_context for a scenario by slug, including realistic metrics, logs, deploy history, and on-call notes. Use as input to your analysis.
Returns the 5-scenario regression library (DB pool exhaustion, OOM deploy, dependency timeout, DNS, cache stampede). Use to test analysis output or as practice incidents.
Detect alert provider and extract structured fields (service, title, symptoms, raw_context) from a webhook JSON payload. Returns null-ish text if not recognized.
Persist a completed incident analysis to the database. The analysis must conform to the structured 9-section schema (call get_output_schema for the spec). Returns the incident_id you can later fetch by URL or via find_similar_incidents.
Output schemas not documented. LLMs cannot predict return structure for any tool. search_kb, find_similar_incidents, get_scenario, list_scenarios all return structured data but no schema is visible in source.
save_incident_analysis accepts 'analysis' parameter as object type with reference to AnalysisSchema, but schema definition is not visible in source code. Inference required; cannot verify structure.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk field is present in metadata but not in tool definition schema. Current MCP spec (2026-07-28) supports tool annotations; should be adopted.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Retrieve relevant excerpts from internal runbooks, postmortems, service catalog, and architecture docs. Uses pgvector + OpenAI embeddings when available, falls back to pg_trgm. Returns top-k chunks ranked by similarity.
Error handling lacks recovery guidance. Tools do not return actionable error messages (e.g., 'Try search_kb() with a different query' or 'Invalid scenario slug; available slugs: db-pool, oom-deploy, ...').
parse_alert_json description states 'Returns null-ish text if not recognized' but does not specify exact return format (null, empty string, error object). Ambiguous for LLM parsing.