Tools for searching engineering runbooks and historical incidents. Use these tools to gather evidence before analyzing an incident.
Three tools with clear verb-noun naming (search_runbooks, search_incidents, read_runbook) and well-structured descriptions (150-200 chars each). All tools have documented input schemas with types and descriptions. Parameters include sensible defaults and constraints (limit 1-5, max_distance bounds). However, output schemas are not formally documented, responses are inferred from code (dict[str, Any]). Error handling is present (ValueError, FileNotFoundError) but lacks recovery guidance for LLMs. No tool annotations (readOnlyHint, idempotentHint) despite all three being read-only operations. Parameter descriptions are adequate but could be more prescriptive about when to use each tool.
Read the complete contents of a specific engineering runbook. Use this tool when a relevant runbook has already been identified and its complete contents are required.
Search historical engineering incidents using semantic similarity. Use this tool to find previous incidents with similar symptoms, affected services, root causes, or resolutions, even when the current incident uses different wording.
Search engineering runbooks using semantic vector similarity. Use this tool when an incident requires troubleshooting guidance, likely causes, investigation steps, remediation guidance, or diagnostic commands.
Output schemas not formally documented. Tools return dict[str, Any] with no schema specification visible to LLMs. Agents cannot predict response structure for downstream planning.
Error messages lack recovery guidance. FileNotFoundError for read_runbook lists available files, which is good, but search_* tools do not guide LLM on empty results or high max_distance values.
No tool annotations despite all three tools being read-only. Missing readOnlyHint would help agents understand these are safe to call without side effects.
Parameter descriptions for search_* tools do not explain when to use search_runbooks vs search_incidents or how to interpret max_distance values (0.95 vs 1.25 defaults differ without explanation).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
read_runbook filename parameter accepts only .md files but does not document available runbooks in the description. LLMs must call the tool blindly or rely on error messages.