Loading ToolBench
MCP server that integrates Semgrep static analysis tool, allowing scanning of code with Semgrep configurations and listing available configurations
Two tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. list_configs has empty input schema ({}), scan has two parameters but lacks descriptions for both. Neither tool documents return structure. Descriptions are present but minimal (10-50 chars). No error handling guidance, no parameter constraints (enums, ranges), no idempotency hints. Tool names follow verb_noun pattern (list_, scan) which is good, but descriptions do not explain WHEN to use each tool or what downstream tools expect. Output is returned as JSON text via NewToolResultJSON(), but the schema structure is not documented in tool definitions.
List available Semgrep configurations
Run Semgrep scan with specified configuration. The results contain the exact file path and line numbers.
list_configs has empty input schema ({}). Tool is discoverable but provides no guidance on parameters or constraints.
scan parameters 'directory' and 'config' lack descriptions in the tool definition. LLMs cannot infer that 'config' is a name (not an ID) or that 'directory' defaults to current dir.
No output schema documented for either tool. scan returns JSON with Result objects (CheckID, Path, Start/End positions, Extra metadata), but this structure is not declared in the tool definition. LLMs cannot plan downstream calls without knowing what fields to expect.
scan description does not explain when to call it vs list_configs, or what prerequisites exist (e.g., must call list_configs first to get valid config names). Descriptions lack WHEN/WHY context.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | 2026-07-28+ | v2 |
No error handling guidance. If semgrep binary is not installed, or config file is invalid, or directory does not exist, the tool returns a generic error. LLMs receive no recovery hints (e.g., 'Install semgrep: brew install semgrep' or 'Call list_configs to see valid config names').