Loading ToolBench
Production LLM quality observatory for Splunk. Agentic closed-loop system that instruments LLM calls, logs them to Splunk via HEC, senses quality regressions through an MCP Server, root-causes via a Splunk hosted model, and remediates with actions like rollback or incident filing.
Single tool 'run_splunk_query' has a reasonable description (127 chars, within 10-1024 range) but critical schema deficiencies. Input schema visible in code shows 'query' and 'earliest' parameters with type and description, but output schema is completely undocumented, LLMs cannot plan downstream actions without knowing what fields are returned. No error handling guidance, no parameter constraints (e.g., SPL syntax validation, time format validation), and no indication of what the tool returns. Tool naming is acceptable (verb_noun: 'run_splunk_query') but lacks clarity on whether it returns raw results, aggregated metrics, or structured quality signals. The description mentions 'quality signals for model versions' but the schema does not reflect this, no output fields documented. Per-tool score capped at 50 due to missing output schema and inferred tool registration (tool definition not explicitly visible in provided code snippets, only referenced in demo.py and live_test.py).
Run SPL via MCP: current-hour avg groundedness vs N-hour baseline. Executes Splunk Processing Language queries to retrieve quality signals for model versions.
Output schema completely undocumented. LLMs cannot infer what fields are returned (e.g., is it a list of rows, a single metric, a time-series?). This blocks downstream tool composition and forces agents to guess at response structure.
No parameter constraints or validation guidance. 'query' parameter accepts free-form SPL strings with no format hints, length limits, or examples of valid queries. 'earliest' parameter lacks format specification (is '-24h' the only valid format? What about '-1d', '2024-01-15T00:00:00Z'?).
No error handling or recovery guidance. If SPL syntax is invalid, if the time range is malformed, or if Splunk is unreachable, the tool provides no actionable error message to guide the LLM's next step.
Tool definition not explicitly visible in provided source code. Tool is referenced in demo.py and live_test.py but the actual MCP tool registration (schema, handler) is not shown. Per-tool score capped at 50 due to inference.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 36 | <=2025-11-25 | v2 |
Description mentions 'quality signals for model versions' but schema does not document what those signals are or how they are structured. Mismatch between description intent and schema reality.