Loading ToolBench
Security Property Graph (SPG) oracle for AI coding agents. Analyzes code for security vulnerabilities including taint paths, attack surfaces, missing sanitizers, and data flow tracing.
VibeGuard SPG defines 8 security-focused taint analysis tools with generally clear, domain-specific descriptions. All tools have descriptions and most have documented parameters. However, significant schema completeness issues, missing parameter descriptions on critical fields, and no visible output schema documentation prevent a higher score. The tool set is well-composed for security analysis (each does one thing), and naming is consistent with verb_noun patterns (query_, get_, find_, trace_, calculate_, check_). But execution against the rubric reveals gaps in parameter constraints, missing type definitions on some inputs, and no documented output structures. However, parameter descriptions are sparse or absent on many required fields, violating the rule 'Every parameter needs a description.' The code snippet shows tool registration in mcp-go, but output schemas (pathResult, response) are only partially visible in the truncated source, making full schema assessment impossible.
[Phase 2] Calculate what security properties could change if a function is modified. Returns all taint paths and trust boundaries affected by changes to the given function.
[Phase 2] Check whether an endpoint enforces authentication before sensitive operations.
Find all locations where tainted data reaches a dangerous sink without a sanitizer in the path. Returns all active taint paths ranked by severity (critical → high → medium → low). This is the primary security audit query — run it after significant code changes.
Return the full attack surface for a module: all untrusted entry points (SourceNodes) and the dangerous sinks reachable from them. Use this before writing new code in a module to understand what attacker-controlled data is already present and where it can travel.
[Phase 2] Get the full security posture of a file: classified nodes, taint status, trust level, risk score.
Output schemas not documented. Code shows partial pathResult and response struct definitions in tools.go, but no complete JSON schema documentation visible for any tool return type. LLMs cannot plan downstream operations or extract fields without documented outputs.
Parameter descriptions missing or minimal on several tools. 'get_trust_boundary_violations' and 'find_missing_sanitizers' have empty input objects {} but no description of what context/constraints apply. 'trace_data_flow' lists 3 params (node_id, symbol, file) but symbol and file lack descriptions, violating 'Every parameter needs a description.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 38 | - | v1 |
[Phase 2] Find code paths that cross trust boundaries without authentication checks.
Find unsanitized taint paths from any SourceNode to a given sink. Returns a deterministic list of source-to-sink paths where attacker-controlled data reaches a dangerous operation without sanitization. Same code + same query always returns the same result — no LLM reasoning in the verification path. Answer: full source-to-sink path with file paths and line numbers, or empty array if no paths exist.
Trace where a variable or expression travels through the codebase. Returns the full provenance of a value: its origin, all transformations, any sinks it reaches, and external calls it participates in. Use this to understand the security implications of a specific data value.
Parameter type constraints not enforced. 'module' param in get_attack_surface is type string with no validation, pattern, or length limits. 'node_id' in trace_data_flow has no documented format. LLMs will pass arbitrary strings without guidance.
No error handling guidance. Tool descriptions state what the tool does but provide no recovery hints. If 'sink' parameter is invalid or no paths found, the tool returns a result, but there is no documented error classification (retryable, user-fixable, fatal) or suggested next steps for the agent.
Response field chaining unclear. Tools reference 'file paths and line numbers' but output field names (e.g., is it 'path', 'file_path', 'filepath', or nested 'source.file'?) are not documented. Agents cannot compose downstream tool calls without knowing the exact response structure.
Tool scope ambiguity on 'module' and 'file' parameters. Description says 'e.g. src/api/ or handlers/' but does not specify: exact path format, whether relative to repo root or absolute, whether trailing slash required, case sensitivity. LLMs will guess and fail.
Missing pagination guidance. Tools like 'query_taint_paths', 'get_attack_surface', and 'find_missing_sanitizers' return lists but do not document limit, offset, or total_count fields. Large codebases could return thousands of paths, risking context window exhaustion.
No tool composition guidance. The rubric expects tools designed so 'common user intents resolve in one call.' Here, to fully audit a codebase, an agent must manually sequence: find_missing_sanitizers → query_taint_paths (per result) → trace_data_flow (per path) → get_security_context (per file). No higher-level 'audit_module' or 'audit_file' wrapper tool is provided.