Loading ToolBench
Semgrep static analysis exposed as MCP tools for AI agents
Server has 5 tools with complete input schemas and descriptions, but quality is uneven. Tool names follow verb_noun convention (scan, list_rulesets, get_finding_details). Descriptions are present but brief (10-60 chars), below the 50-200 char LLM-optimized baseline. Parameters lack detailed constraints (e.g., rulesets array has no enum of valid values, max_findings has no min/max bounds). Output schemas are partially documented in code but not formally declared. Error handling is minimal, _run_semgrep raises RuntimeError with truncated stderr, offering no recovery guidance. No tool annotations (readOnlyHint, etc.). Composition is reasonable: tools are single-purpose and chainable (scan → get_finding_details). Security: tools accept file paths without validation, risking path traversal; no permission gates or audit logging.
Scan only findings introduced since base_ref. Requires repo to be a git repo.
Rule metadata + remediation hint for a specific rule ID.
Built-in rulesets available without a Semgrep account.
Run the default scan (p/security-audit + p/secrets) over a repo path.
Scan with one or more specific rulesets.
Parameter constraints missing: rulesets array has no enum of valid values; max_findings lacks min/max bounds (e.g., 1 - 1000); base_ref has no format hint (git ref syntax). LLMs will hallucinate invalid values.
Descriptions are too brief (10 - 60 chars, below 50 - 200 char baseline). 'Run the default scan' and 'Scan with one or more specific rulesets' lack context on WHEN to use each tool, what they return, and prerequisites (e.g., 'baseline_scan requires a git repo').
Error handling is non-actionable. _run_semgrep raises RuntimeError with truncated stderr (500 chars max). LLM receives no guidance on retry, recovery, or next steps. get_finding_details returns bare error string instead of structured error with rule_id and actionable hint.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
No input validation on repo_path. Accepts any string; no check for path traversal (e.g., '../../../etc/passwd'). subprocess.run with user-supplied path is a command injection risk if semgrep interprets special chars.
Output schemas not formally documented. Code shows _normalize_findings returns list of dicts with fields (rule_id, severity, cwe, owasp, file, line_start, line_end, message, fix, references), but no JSON Schema or type hints in tool signatures. LLM cannot infer downstream field availability.