Loading ToolBench
MCP server and CLI for providing AI coding agents with live production context. Exposes diagnostic tools for fetching recent errors, deployments, service health, and automated root cause diagnosis.
DevContext provides 4 tools with complete input schemas and descriptions, but quality is uneven. Tool names follow verb_noun convention (get_*, diagnose). Descriptions are present but generic, they explain WHAT without sufficient WHEN/WHY context for LLM selection. Parameters have types and descriptions, but lack constraints (enums, ranges, patterns). Output schemas are not documented, LLMs cannot predict return structure. Error handling is absent from descriptions. The 'diagnose' tool conflates multiple responsibilities (calls 3 other tools internally), violating single-responsibility principle. No security guidance (e.g., file path validation, secret handling). Overall: functional but below production baseline.
Run a full automated diagnosis for a service and identify the likely root cause. Calls get_recent_errors, get_recent_deploys, and get_service_health, then synthesises a plain-English "likely_cause" statement. You can either pass `service_name` to use built-in demo data, or supply explicit paths (`log_path`, `deploys_path`, `health_path`) to point at real log files/folders and metric JSON files.
Retrieve the most recent deployments for a service, sorted newest-first. Accepts an optional `deploys_path` parameter pointing directly to a JSON file containing deployment records. If None, falls back to `mock_data/deploys.json`.
Fetch and summarise recent ERROR and WARN log lines for a service. Accepts an optional `log_path` parameter pointing directly to a log file or to a directory containing multiple `.log` / `.txt` files (which will be read and concatenated). If `log_path` is not provided, it falls back to looking up `mock_data/<service_name>.log`. If `use_llm_extraction=True` is passed, it uses Groq Llama 3.1 8B Instant (via extraction.py) to parse arbitrary/unstructured log formats into JSON.
Return current health metrics for a service, with a plain-English flags list. Accepts an optional `health_path` parameter pointing directly to a JSON health metric file. If None, falls back to `mock_data/health.json`.
Output schemas not documented. LLMs cannot predict return structure (fields, types, nesting). Descriptions mention 'dict' but omit field names, types, and examples.
'diagnose' tool violates single-responsibility principle. It calls get_recent_errors, get_recent_deploys, and get_service_health internally, then synthesizes results. This should be orchestrated by the agent, not the tool.
Parameters lack constraints. 'minutes' (integer) has no min/max bounds, LLM could pass negative or absurdly large values. 'limit' (integer) unbounded. No enums for 'service_name' even though examples suggest a fixed set.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
Descriptions lack WHEN/WHY guidance. 'Fetch and summarise recent ERROR and WARN log lines' explains WHAT but not when to call this vs diagnose, or what the LLM should do with results. No dependency hints (e.g., 'Call this first to identify error patterns').
No error handling guidance. Code catches exceptions (FileNotFoundError, ValueError) but descriptions do not explain recovery steps. LLM receives raw errors with no actionable next steps.