Loading ToolBench
AI incident responder with a custom MCP server (LangGraph + FastAPI). Provides tools for querying metrics, searching logs, and executing remediation runbooks.
SREcopilot exposes 5 tools with clear verb-noun naming (query_metric, search_logs, list_runbooks, get_runbook, execute_runbook). All tools have descriptions and input schemas with typed parameters. However, descriptions are terse (10-60 chars), parameter descriptions lack detail on constraints/formats, output schemas are not documented, and error handling is minimal. The execute_runbook tool lacks a confirmation guard in the MCP layer despite claiming one in the description. No tool annotations (readOnlyHint/destructiveHint) are present. Schemas are present but sparse, most parameters lack constraints (enums, ranges, patterns). This is typical community-grade work: functional but not production-ready.
Execute a runbook. Refuses to run unless confirm=True (defense in depth).
Fetch the full details of a runbook by id.
List the available remediation runbooks (id, title, risk).
Query a metric (e.g. error_rate, p99_latency) for a service over a recent window.
Search recent logs for a service. Use query='*' to fetch everything.
Output schemas not documented. Tools return JSON strings but LLMs cannot infer field structure. query_metric returns {metric, service, window_minutes, points[], summary} but this is not declared in the tool definition. Agents must guess what fields exist.
Parameter descriptions lack constraints. 'metric' param has no enum or pattern (e.g., 'one of: error_rate, p99_latency, cpu, memory'). 'window_minutes' has no range (1-1440?). 'limit' in search_logs has no bounds. LLMs will hallucinate invalid values.
Tool descriptions are too brief (10-60 chars). 'Query a metric...' and 'Search recent logs...' lack context on WHEN to use each tool, what data they return, or dependencies. Baseline is 194 chars; these are 30-50% of that.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | <=2025-11-25 | v2 |
execute_runbook claims 'Refuses to run unless confirm=True' but the MCP wrapper (mcp_server.py line ~60) does NOT validate confirm. It only returns a JSON error if confirm=false, but the tool still executes. The defense-in-depth claim is false.
No tool annotations. execute_runbook is destructive (IRREVERSIBLE risk) but has no destructiveHint annotation. query_metric and search_logs are read-only but have no readOnlyHint. Agents cannot infer safety properties from the tool definition.