Loading ToolBench
An agentic SOC analyst that investigates security questions by querying Splunk through the official Splunk MCP Server, with a deterministic 3-layer validator that gates findings to prevent AI hallucination. Code, not the model, decides what's confirmed.
Single tool 'splunk_run_query' has minimal definition quality. Description is present but generic (47 chars, below 50-200 char baseline for LLM optimization). Input schema visible but extremely sparse: only 'query' parameter with basic string type, no format constraints, no examples of valid SPL syntax, no guidance on query limits or timeout behavior. No output schema documented, LLMs cannot predict what fields to expect from Splunk results. No error handling guidance (what happens on invalid SPL, timeout, permission denied?). Tool name is acceptable (verb_noun pattern) but lacks context about what 'run' means in Splunk domain. No parameter validation rules, no pagination support despite Splunk queries often returning large result sets. Security concern: no mention of credential handling or query injection prevention.
Execute a Splunk search query (SPL) and return results
No output schema documented. LLMs cannot predict result structure (fields, types, pagination). Splunk queries return variable schemas depending on SPL, tool must document expected output format or at minimum describe common fields (timestamp, source, host, event).
Input parameter 'query' lacks format constraints and validation guidance. No mention of SPL syntax rules, query length limits, timeout behavior, or examples of valid queries. Description should state: 'SPL query string (max 10000 chars, no raw pipe commands, timeout 60s)' and provide a valid example.
No error handling guidance. Tool description does not explain what happens on invalid SPL, authentication failure, timeout, or permission denied. Error responses must guide LLM recovery: 'Invalid SPL syntax at token X. Try search_splunk_syntax() for help.' or 'Permission denied, you lack read access to index. Contact admin.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | 2026-07-28+ | v2 |
No pagination or result limiting. Splunk queries can return millions of events. Tool must accept limit and offset/cursor parameters and document max result size (e.g. 'Returns max 1000 events; use offset for pagination'). Without pagination, large result sets blow context window.
Tool description (47 chars) is below LLM-optimization baseline (50-200 chars). Expand to explain: WHAT it does (executes SPL queries), WHEN to use it (after identifying relevant index/sourcetype), WHAT it returns (structured events with fields), and any prerequisites (valid Splunk credentials, index access).