Loading ToolBench
A standalone, pluggable normalizing meta-layer over your monitoring tools. Provides read-only discovery and query tools for signals, dashboards, metrics, incidents, logs, health status, cluster state, cost, and SLOs.
Panoptes defines 11 tools with consistent naming (verb_noun pattern: describe_, list_, get_, search_, compare_) and reasonable descriptions (avg ~150 chars). All tools are READ_ONLY, reducing risk. However, critical gaps exist: input schemas are INFERRED from type hints in docstrings rather than explicitly registered with JSON Schema; parameter descriptions are minimal (many lack detail on format/constraints); output schemas are undocumented; no error handling guidance; no pagination support despite tools returning potentially large result sets. The codebase is well-structured (fastmcp, strict mypy, good test coverage), but the MCP tool definitions themselves lack the rigor expected for production agent use.
One metric's series across every enabled env. Maps each ENABLED env to its resolved series for `metric` over `window`; carries a per-env marker for any env whose query could not be answered (a partial result).
The 'one thing to look at' rollup: per-source reachability (an unreachable source is INCLUDED, marked unreachable, never omitted) + the open incident count.
List the environments, configured sources + capabilities, metrics, dashboards.
A point-in-time snapshot of one env's kubernetes cluster, read from the STORE. Rendered from the stored `panoptes_k8s_*` gauges (the two-faces-one-store parity — the Grafana Kubernetes dashboard renders the SAME series), NOT from a live cluster call.
A cost snapshot for one env over a window, read from the STORE's cost gauges. Rendered from the stored `panoptes_cost_*` gauges (the two-faces-one-store parity — the Cost dashboard renders the SAME series), NOT a live Cost Explorer call.
Input schemas are inferred from type hints in docstrings, not explicitly registered with JSON Schema. Cannot verify actual schema structure, types, or constraints at registration time.
Output schemas are completely undocumented. Tools return complex types (QueryContext, DashboardPack, series data) but LLMs have no visibility into response structure, field names, or types. Agents cannot plan downstream calls or extract data reliably.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
Execute one dashboard's panels for `env` and return title + PromQL + series. Resolves the pack by id, reads its dashboard JSON, and for each panel target substitutes the `$env` template variable with `env`, executes the resulting PromQL against the store, and attaches the resolved series.
Evaluate one SLO for one env. Measures attainment over the SLO's window and computes error budget remaining.
Return the dashboard catalog (core + injected consumer packs) as summaries.
A thin PromQL passthrough to the store. It is the only tool that reads the store directly with caller-supplied parameters, so it is the natural surface for a `passthrough`-store `CapabilityError`.
Fetch the requested env's sources, filter to the relevant signal kind, and return the matching signals.
Fetch the requested env's sources, filter to the relevant signal kind, and return the matching signals.
Parameter descriptions lack actionable detail. 'env' is described as 'Environment name or all for fan-out' but no enum of valid env names, no format constraints, no guidance on what 'all' returns. 'window_minutes' lacks min/max bounds. LLMs cannot validate inputs or self-correct.
No pagination support. Tools like list_dashboards and search_incidents may return unbounded result sets. No limit, offset, page_size, or next_cursor parameters. Large results will blow context windows and degrade LLM reasoning.
No error handling guidance. Tools are READ_ONLY (low risk), but no documentation of failure modes, recovery steps, or what to do if a query fails, an env is unreachable, or a dashboard_id does not exist. Agents have no recovery path.