Loading ToolBench
Open security scanner and self-hosted control plane for AI, MCP, and cloud infrastructure. CVEs, blast radius, credential exposure, runtime enforcement.
The server implements 8 tools with reasonable domain coverage (correlation, diffing, triage, ticketing). All tools have descriptions (10 - 500 chars). Input schemas are present with type declarations and constraints. However, there are significant gaps: (1) output schemas are NOT documented anywhere in the provided source, callers cannot predict response structures; (2) parameter descriptions are inconsistent, some are detailed (e.g., graph_correlate), others are minimal (e.g., example_asset_posture); (3) error handling is not visible in the tool definitions, no recovery guidance, no error categorization; (4) naming is generally good (verb-noun style), but tool names like 'graph_correlate' and 'example_asset_posture' are vague about what they return; (5) three tools (graph_correlate, diff, findings_triage, create_ticket) are DESTRUCTIVE but their descriptions could be clearer about side effects and idempotency semantics. The server accepts natural IDs (names, tenant scopes) which is good. Overall, solid parameter definitions but weak on outputs and error guidance.
File an ITSM ticket for a finding through a stored connection. Connect-once: auth and the ITSM base URL come only from the stored, encrypted connection — no credential or link is passed here. Requires an admin operator + ticketing:write scope. Idempotent per finding.
Compare a fresh scan against a baseline to find new and resolved vulns. Runs a new scan, then diffs it against the provided baseline (or the latest saved report). Shows new vulnerabilities, resolved ones, and changes in the package inventory. Not read-only: this persists the fresh scan to report history and may prune older saved reports, so it is annotated as a (destructive) write.
Return a small metadata-only posture payload for an operator asset.
Record a tenant-scoped finding triage decision to the exception store. Writes the same entry as the REST POST /v1/findings/triage endpoint. Requires an admin operator + findings:write scope. A not_affected decision requires an OpenVEX justification.
Create one bounded, provenance-rich correlation snapshot.
No output schemas documented for any tool. Callers (LLMs, agents) cannot predict response structure, required fields, or nested object shapes. This blocks effective tool chaining and response parsing.
Destructive tools (graph_correlate, diff, findings_triage, create_ticket) lack idempotency contracts. They accept idempotency_key or finding_id parameters, but do not document whether repeated calls with the same key are safe or what the retry semantics are. Agents need clear idempotent guarantees.
Error handling not visible in tool definitions. No recovery guidance, error categorization, or actionable messages. If a tool call fails (e.g., correlation timeout, invalid vulnerability_id), the descriptions do not explain what the LLM should do next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 67 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Read run state, receipts, freshness, conflicts, and analysis bounds.
List vulnerability exceptions with optional status filtering.
Refresh a filed ticket's status from its ITSM through the connection. Requires an admin operator + ticketing:write scope. Resolves auth and endpoint from the stored connection only.
Parameter constraints incomplete. Several tools have loose descriptions without explicit min/max or enum declarations: diff's 'baseline' is an arbitrary JSON object (no schema or structure doc); findings_triage's 'decision' has a description but no enum constraint in schema; create_ticket's 'issue_type' is free-form. This invites invalid values.
example_asset_posture has vague description ('Return a small metadata-only posture payload') and unclear naming. What does 'posture' mean in this context? What fields are returned? Is this for security posture, compliance, or something else? LLMs cannot reliably infer when to call this tool.
diff tool naming is ambiguous. Does 'diff' compute the difference, report findings, or trigger a scan? Description clarifies (run new scan, compare to baseline), but the tool name alone does not convey the action. Consider 'scan_and_diff_baseline' or 'diff_vulnerability_reports'.
No batch variants offered. If an agent needs to triage 50 vulnerabilities, it must call findings_triage 50 times. A batch_triage_findings or findings_triage accepting an array would reduce token overhead and latency.
Operator role and scope parameters (operator_role, operator_scopes) appear on destructive tools with defaults ('viewer', empty string). Default role 'viewer' for a write action is confusing, should this be 'admin'? Defaults that weaken security invite misuse.