Loading ToolBench
Adversarial vulnerability scanning for local agent councils via MCP. Runs Giskard security scans against MCP servers to detect prompt injection, information disclosure, and other LLM vulnerabilities.
Server has 6 tools with clear action verbs (run_, list_, get_) and reasonable descriptions (100-200 chars). All tools have input schemas with typed parameters and descriptions. However, output schemas are not documented in the source code, parameter descriptions lack detail on constraints/formats, and error handling guidance is absent. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles. Naming is good but descriptions could be more LLM-optimized with dependency hints and recovery guidance.
Return the scan record plus file metadata for one HTML report.
Return one persisted scan record with its issues and report path.
Return the current status of the Giskard Red-Team Node.
Discover running MCP servers in the fleet port range (10700-11500). Probes each port for health endpoints and returns a list of active servers with their URLs and service names.
List generated HTML vulnerability reports on disk (newest first).
Run an automated Giskard adversarial scan against a fleet MCP server. Connects to the target MCP server via HTTP SSE, discovers its tools, wraps the server as a Giskard text_generation model, and runs Giskard's adversarial detectors (optionally limited to the given profiles). The eval LLM is the saved Settings model (or auto-detected default).
Output schemas not documented. LLMs cannot plan downstream calls or extract required fields (e.g., what fields does run_fleet_scan return? Does it include issue counts, severity levels, remediation steps?).
Parameter descriptions lack actionable constraints. 'profiles' accepts 'comma-separated scan profiles' but does not list valid profile names (prompt_injection, information_disclosure, etc.). LLMs cannot validate input without enums or explicit allowed values.
No error handling guidance. run_fleet_scan may fail if the target MCP server is unreachable, misconfigured, or returns unexpected responses. No recovery hints (e.g., 'Check that the MCP server is running at the given URL').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Tool annotations missing. run_fleet_scan is destructive (initiates scans, writes reports) but lacks destructiveHint. list_* and get_* tools are read-only but lack readOnlyHint. Agents cannot reason about side effects.
Parameter 'agent_description' in run_fleet_scan has empty default but no guidance on when it is required. Description says 'Used by Giskard to generate targeted attacks' but does not explain impact of omitting it or format expectations.