Loading ToolBench
A hardened MCP server that gives an AI agent a code security scanner, built so the tools cannot be turned against the host, with its own red-team suite proving the trust boundaries hold.
Two well-named, security-focused tools with clear descriptions and proper input schemas. scan_code and scan_path follow verb_noun convention and have 150+ char descriptions explaining WHAT, WHEN, and constraints. Both parameters are typed (string) with descriptions. However, output schemas are not formally documented in the tool definitions, only inferred from code comments. Error handling is structured (GuardRejection → _error dict) but lacks actionable recovery guidance for LLMs. No tool annotations (readOnlyHint, idempotentHint) despite both being read-only operations.
Scan a snippet of code for leaked secrets and insecure patterns (OWASP Top 10). Returns findings sorted by severity. No filesystem access. `filename` is an optional label used for reporting.
Scan a single source file for leaked secrets and insecure patterns. `path` must be relative to the server's workspace root; paths that escape the workspace are refused.
Output schema not formally documented in tool definitions. LLMs cannot see the structure of findings, summary, or target fields without reading implementation code.
Error responses lack actionable recovery guidance. GuardRejection errors return {ok: false, error, code} but do not suggest next steps (e.g., 'path must be relative; try removing leading /'). LLMs cannot self-correct.
Tool annotations missing. Both tools are read-only and idempotent but lack readOnlyHint and idempotentHint annotations. Agents cannot infer safety properties without explicit hints.
Parameter descriptions lack format/constraint details. 'code' param says 'code snippet' but does not state size limit (MAX_TEXT_BYTES=5MB inferred from guard.py). 'path' param does not explain relative-path requirement clearly enough for LLM to avoid absolute paths.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2025-06-18+ | v2 |
No pagination or result-limiting guidance in descriptions. If findings list grows large, LLMs receive unbounded output. Descriptions should state 'Returns up to N findings sorted by severity' and explain how to handle overflow.