Loading ToolBench
A Hybrid-AI Security & Quality Sentinel for the MCP Era
ShieldAgent has 7 tools with complete input schemas and descriptions, but quality is uneven. All tools have descriptions (10-100 chars, mostly adequate), and all parameters are typed with descriptions. However, output schemas are entirely undocumented, LLMs cannot predict what fields to expect from responses. Error handling is present but generic (returns error strings without recovery guidance). No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles. Naming is verb-forward and clear (scan_*, list_*, read_*, audit_*, safe_write_*, restore_*, scan_*), following conventions well. Parameter defaults are sensible (directory='.', use_ollama=false). Security controls (path validation, prompt injection firewall) are implemented but not exposed in tool descriptions, so LLMs cannot reason about constraints. Overall: solid foundation, but missing output documentation and error recovery patterns that would elevate this to 70+.
Performs a deep Gemini-powered security audit on a specific file.
Lists files and directories in the specified path.
Reads the full content of a specified file.
Restores a file from its .bak backup. Used to rollback a flawed remediation.
Safely writes or updates a file with a mandatory justification. Used by remediation agents to fix security vulnerabilities.
Scans specified directory for sensitive data leaks (PII, Secrets).
Scans CI/CD pipelines and Dockerfiles for supply chain vulnerabilities. Checks for mutable tags in GitHub Actions and permissive scopes.
No output schemas documented. LLMs cannot predict response structure (fields, types, pagination). All 7 tools return unstructured strings; no typed response objects.
Error responses lack recovery guidance. Errors return plain strings (e.g. '❌ Error: Path not found') without suggesting next steps or categorizing as retryable/user-fixable/fatal.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools have clear risk profiles (READ_ONLY, WRITE, REVERSIBLE) but LLMs cannot see them in schema.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Path traversal and prompt injection protections are implemented but not documented in tool descriptions. LLMs cannot reason about constraints (e.g. 'paths must be within working directory').
scan_for_secrets and scan_supply_chain return unstructured report strings. No pagination, no structured issue objects with fields (file_path, line_number, rule_name, severity). Large scans will produce massive unstructured output.