Loading ToolBench
MCP service for validating Suricata rules against pcap files
Four tools with complete schemas and basic descriptions, but significant gaps in parameter documentation and error guidance. Tool names follow verb_noun pattern (validate_*, test_*) which is good. Descriptions are present but generic (avg ~100 chars, below 194-char baseline). Critical issue: parameters lack descriptions in most cases. No output schema documentation. No error handling guidance. No tool annotations (readOnlyHint present in metadata but not in tool definitions). Composition is reasonable, tools are single-purpose and chainable.
Test a Suricata rule against a pcap file to verify if the rule generates expected alerts. Returns alert information including signature_id matching.
Test a Suricata rule file against a pcap file to verify if the rules generate expected alerts.
Validate Suricata rule syntax. Checks if the rule has any syntax errors.
Validate Suricata rule file syntax. Checks if the rule file has any syntax errors.
Parameter descriptions missing or minimal. 'rule_content', 'pcap_path', 'expected_sid', 'rule_path', 'expected_sids' have only 1-2 sentence descriptions. LLMs cannot infer parameter semantics from names alone, need explicit format, constraints, and examples.
No output schema documentation. Tools return JSON with fields like 'valid', 'error_message', 'alerts', 'matched', but LLMs have no formal schema to understand structure, types, or required fields. Downstream tool chaining is blind.
No error handling guidance. Exception handler returns generic 'Error executing {name}: {str(e)}' with no recovery hints. LLMs cannot distinguish retryable errors from user-fixable ones. No actionable error messages.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | <=2025-11-25 | v2 |
Tool annotations missing from definitions. Risk metadata (READ_ONLY) is present in evaluation metadata but not exposed in tool inputSchema via readOnlyHint. LLMs cannot see which tools are safe to retry or have side effects.
No input validation or constraint documentation. Parameters like 'rule_content' and 'pcap_path' accept free-form strings with no format hints, length limits, or validation rules. LLMs may pass invalid paths or malformed rules without guidance.