Loading ToolBench
Vigil SOC - Security Operations Center platform with MCP tool integration for threat intelligence, detection rules, and security analysis
Vigil SOC exhibits significant definition quality gaps across its 29 tools. While tool names follow verb_noun conventions (e.g., otx_check_ip, create_case, list_findings), descriptions are minimal and many lack actionable context. Input schemas are present but sparse: most tools have only 1-3 parameters with minimal type information. Critical issues: (1) Parameter descriptions are entirely absent, LLMs cannot infer what 'severity', 'status', 'sort_by' mean without explicit guidance. (2) Output schemas are undocumented, no tool declares what fields it returns, forcing LLMs to guess. (3) Error handling is not visible in tool definitions. (4) Many tools (e.g., recall_memory, reconstruct_run, lint_detections) have empty input schemas {}, making their purpose opaque. (5) Tools like update_case accept only case_id as required, leaving other updateable fields ambiguous. Baseline: 94% of A+ tools have parameter descriptions; 100% document return types. Vigil meets neither.
Add a finding to an existing case
Add a resolution step to a case
Analyze detection coverage for security findings
Get Any.Run sandbox analysis report
Search Any.Run by file hash
Create a new security case
Get a specific case by ID
Parameter descriptions entirely absent. Tools like list_findings, search_findings, and update_case expose parameters (severity, status, sort_by, sort_order) with no descriptions. LLMs cannot infer what 'severity' means (finding severity? case severity?) or what values are valid without explicit guidance.
Output schemas undocumented. No tool declares what fields it returns. E.g., get_finding returns a finding object, but LLMs don't know if it includes case_id, severity, timestamps, or raw API response. This forces LLMs to guess and breaks tool chaining.
Empty input schemas on 8 tools (recall_memory, analyze_coverage, search_detections, identify_gaps, get_coverage_stats, get_detection_count, lint_detections, reconstruct_run). Schema is {} with no properties. LLMs cannot determine if these tools accept parameters or what they do.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | <=2025-11-25 | v2 |
Get statistics about detection coverage
Get count of available detections
Get a specific finding by ID
Get statistics about findings grouped by severity, data source, and status
Get MITRE ATT&CK technique statistics from findings
Identify gaps in detection coverage
Lint detection rules for quality issues
List security cases with optional filtering by status and severity
List completed threat hunts with optional time range filtering
List security findings with pagination and filtering
Look up threat indicators in the local indicator database
Check domain in OTX
Check file hash in OTX
Check IP in AlienVault OTX
Propose hunts based on recent threat feed indicators
Read a skill from the skill library
Recall episodic memory from previous hunts
Reconstruct a hunt run with detailed analysis
Replay a completed hunt with optional decision filtering
Search detection rules by criteria
Search security findings with query and filtering
Update case properties
Ambiguous parameter names without type suffixes. lookup_indicators accepts both 'values' (array) and 'value' (string) with no description of when to use each. LLMs will guess wrong and pass both, causing ambiguous calls.
update_case accepts only case_id as required parameter, with no description of what fields can be updated. LLMs cannot determine if they can update status, severity, title, or description without trial and error.