Loading ToolBench
Static security scanner for MCP servers and tools.
The server defines three security scanning tools with reasonable naming and descriptions. Tool names follow verb_noun convention (list_rules, scan_project, get_recommendations) which is appropriate. Descriptions are present and moderately detailed (120-250 chars). However, there are critical gaps in parameter descriptions, schema completeness for complex objects, and error handling guidance. The scan_project tool has the most comprehensive schema with proper enums and types, but input descriptions lack constraint details (e.g., what valid 'languages' values are, file path format requirements). Output schemas are undocumented, the code returns MCPActionResult objects but the LLM has no visibility into the structure of 'summary', 'findings', or 'blocking' fields. Error handling is minimal, no guidance on what to do if a scan fails or a rule is not found. The get_recommendations tool accepts optional context parameters with type 'object' but no structure definition, forcing LLMs to guess at valid keys.
Get security best practices and recommendations, optionally scoped to a specific rule or file path.
List all available security rules with their metadata, including rule ID, name, category, severity levels for local and shared modes, and OWASP mappings.
Scan a project directory or .zip archive for security vulnerabilities using static analysis rules. Supports multiple output formats (JSON, SARIF, Markdown) and configurable scan modes (local or shared).
Output schemas not documented for LLM visibility. Code returns MCPActionResult with 'summary' (dict), 'findings' (list), and 'blocking' (bool) fields, but tool definitions do not describe these fields. LLMs cannot plan downstream operations without knowing output structure.
Parameter descriptions lack constraint details. 'languages' parameter lists enum values in schema but description does not explain what languages are supported or when to use the filter. 'config_path' accepts 'nullable' without explaining what config file format is expected. File path parameters (path, file_path, config_path) have no format or validation guidance.
Error handling provides no recovery guidance. If scan_project fails (e.g., invalid path, unsupported file type), the implementation raises ValueError but the LLM has no guidance on what to do next. If get_recommendations is called with a non-existent rule_id, it silently returns generic best practices instead of indicating the rule was not found.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 44 | - | v1 |
get_recommendations 'context' parameter is type 'object' with no schema or description of valid keys. LLMs cannot determine what context fields to pass. This violates the constrained-input pattern.
scan_project description does not specify output format defaults or implications. If agent omits 'output_format', what is returned? SARIF output is specialized for CI/CD; returning it by default when Markdown would be more readable to LLMs wastes tokens.
No pagination support. Tools that return lists (list_rules returns all rules, scan_project returns all findings) do not accept limit or offset parameters. Scanning a large project could return hundreds of findings, exhausting context window.