Loading ToolBench
Autonomous SOC triage system for phishing email investigation with RBAC-protected MCP gateway, threat intelligence integration, SIEM/EDR querying, and AI-driven verdict generation
Server has 5 tools with basic schemas and descriptions, but significant gaps limit production readiness. Tool names follow verb_noun convention (get_, check_, query_, save_). Descriptions exist but are brief (avg 95 chars, below 194-char baseline). Input schemas are present with type definitions, but parameter descriptions are minimal or missing context. The save_investigation_report tool has the most complete schema with 8 required fields and proper typing. No output schemas documented. Error handling is absent, no recovery guidance, retryability classification, or actionable error messages. No tool annotations (readOnlyHint/destructiveHint). Composition is reasonable: tools chain logically (get_email → check_threat_intel → query_splunk → query_endpoint → save_report), but no pagination, no batch operations, and no documented output structure.
Queries threat intelligence for a domain, IP, URL, or file hash. indicator_type must be one of: 'domain', 'ip', 'hash', 'url', 'filename'.
Retrieves extracted indicators from a reported suspicious email via Mailpit using its internal Mailpit ID. Always run this first.
Queries EDR logs in Splunk to check for suspicious process execution on a specific machine. Run this ONLY IF a user has clicked a malicious link.
Queries the SIEM (Splunk) to see if any users clicked a specific URL. Use this to determine the blast radius.
Saves the final investigation verdict and summary into the local case management database. Run this ONLY as the final step.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields without knowing response structure.
Parameter descriptions are minimal or absent. 'indicator_type' in check_threat_intel lists enum values in description but lacks context on when to use each type. 'ip_address' in query_endpoint_activity has no description at all.
No error handling or recovery guidance. Tools provide no indication of retryability, user-fixable errors, or actionable next steps on failure.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | <=2025-11-25 | v2 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). save_investigation_report is destructive (WRITE) but lacks annotation; read-only tools lack annotation.
Tool descriptions lack WHEN/WHY context. 'Queries threat intelligence for a domain, IP, URL, or file hash' states WHAT but not WHEN to call it vs other tools or what to do with results.