Loading ToolBench
A SOC threat triage agent that investigates security incidents using MCP tools for threat intelligence, firewall logs, user activity, and asset details. Integrates with Pydantic AI, OpenAI, and Logfire for structured incident reporting.
Six tools with basic verb-noun naming (get_, search_, lookup_). All have descriptions (10-200 chars, within baseline range). Input schemas present with type definitions and parameter descriptions. However, output schemas are NOT documented, tools return JSON strings without declaring field structure. No error handling guidance. No parameter constraints (enums, ranges). Descriptions lack WHEN-to-use context and dependency hints. Mock implementations return hardcoded data, raising questions about real-world applicability. Baseline: 194 chars avg description, 72 chars avg param annotation, this server meets description length but lacks depth and actionability.
Look up asset details by IP address.
Get active threat intelligence feed for the past N days. Use this to check if an incident matches known active campaigns.
Get recent activity for a specific user account. Use when you need to understand what a user has been doing.
Check IP address against threat intelligence database.
Search firewall logs filtered by source IP and/or destination port. Use when you need raw firewall evidence for a specific IP or port.
Search SIEM logs for events matching the query.
Output schemas not documented. Tools return JSON strings without declaring field structure (e.g., get_threat_feed returns 'active_campaigns', 'trending_malware', 'top_attack_vectors' but LLM cannot infer these fields). LLMs cannot plan downstream calls or extract data reliably.
No parameter constraints (enums, ranges, patterns). 'hours' accepts any integer with no bounds, LLM could pass 999999 or negative values. 'source_ip' and 'ip_address' accept free-form strings with no validation hints. Invites invalid input.
Descriptions lack WHEN-to-use context and dependency hints. 'lookup_ip_reputation' says 'Check IP address against threat intelligence database' but does not explain when to call it vs get_asset_details, or that it requires an IP (not a hostname). Ambiguous tool selection.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
No error handling guidance. Tools return mock data (hardcoded IPs, fake logs) with no indication of failure modes. If a real IP lookup fails, LLM receives no recovery hint. No categorization of retryable vs fatal errors.
Mock implementations return hardcoded data (e.g., 'lookup_ip_reputation' checks if IP starts with '10.' or ends with '.99' to decide maliciousness). Real threat intelligence integration missing. Raises questions about production readiness.