Offline SOC investigation framework exposing security alert analysis, MITRE ATT&CK mapping, and query generation tools over the MCP protocol
Six tools with inconsistent quality. All have descriptions (good), but schemas are partially visible and parameter documentation is sparse. investigate_alert has a structured AlertInput schema; map_mitre and generate_queries accept AlertObservables but lack explicit type definitions in visible code. parse_wazuh_alert, parse_linux_auth_log, and analyze_linux_auth_activity accept only file_path (string) with minimal validation. No tool declares output schemas. Error handling is absent, no recovery guidance, no categorization. Naming is verb-forward (investigate_, map_, generate_, parse_, analyze_) but generic. No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Overall: functional but below production baseline.
Analyze parsed Linux SSH/auth log activity from a local telemetry sample and produce SOC-style triage guidance. Reuses parse_linux_auth_log parsing logic and generate_investigation_runbook. No SSH commands or API calls. Returns investigation_runbook alongside risk scoring and parsed activity.
Generate offline example investigation pivots for common SIEM/EDR languages. Returns qradar_aql, sentinel_kql, defender_advanced_hunting_kql, and opensearch_dql templates. These are illustrative pivots, not live queries.
Investigate a structured security alert offline. Returns an analyst-ready investigation package with summary, severity assessment, MITRE mappings, recommended queries, next steps, detection opportunities, confidence, and limitations. Does not call live SIEM/EDR APIs.
Map an alert type to a deterministic MITRE ATT&CK technique. Offline rule-based mapping only — not threat-intel enriched. Returns technique_id, technique_name, tactic, confidence, and rationale.
Parse a local Linux SSH/auth log sample and extract failed and successful login events. Reads the file from the lab directory only (no SSH commands, no API calls).
No output schemas documented. LLMs cannot predict response structure or plan downstream tool calls. investigate_alert returns dict[str, Any]; map_mitre returns dict[str, Any]; generate_queries returns dict[str, list[str]]. Actual field names and types are invisible to the agent.
File-based tools (parse_wazuh_alert, parse_linux_auth_log, analyze_linux_auth_activity) accept only file_path: string with no validation schema. No min/max length, no regex pattern, no enum of allowed paths. Path traversal risk: LAB_ROOT check exists in code but not declared in schema. LLM cannot reason about valid inputs.
No error handling or recovery guidance. parse_wazuh_alert returns 'Error: File path is outside the allowed lab directory' and 'Error: File not found' as plain strings. No categorization (retryable vs user-fixable vs fatal). No actionable next steps. LLM cannot self-correct.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2025-06-18+ | v2 |
Parse a Wazuh alert JSON file and extract SOC-friendly observables.
AlertObservables parameter in map_mitre and generate_queries lacks inline type documentation. Code shows it is a Pydantic model (AlertObservables) but schema visibility is incomplete. LLM cannot determine which fields are required or optional.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All tools are read-only (no state mutation), but this is not declared. Agents cannot optimize retry logic or understand safety guarantees.