Loading ToolBench
Deterministic audit of agent skills and prompts against 8 supply-chain attack patterns. No LLM, no network.
Four well-named, read-only audit tools with clear descriptions and complete input schemas. Tool names follow verb_noun pattern (audit_*). Descriptions are substantive (100-200 chars) and explain purpose and use case. Input parameters have types and descriptions. However, output schemas are not formally documented in the code, findings structure is inferred from implementation. Error handling is present but minimal (no recovery guidance). No tool annotations (readOnlyHint) despite all being read-only. Missing parameter constraints (e.g., pattern field should have min/max length). Overall solid for a security audit domain but lacks polish for production LLM integration.
Audit every matching file in a directory (default *.md) against the 8 patterns. Returns per-file verdicts and a summary. Use to scan a downloaded skills collection before installing anything.
Audit a file on disk (SKILL.md / AGENTS.md / CLAUDE.md / system prompt) against the 8 malicious-skill patterns. Returns verdict + line-numbered evidence.
Audit arbitrary text (a SKILL.md, AGENTS.md, system prompt, or code snippet) against the 8 malicious-skill patterns. Returns verdict + evidence per finding. Use before installing or running anything you didn't write.
Return the full catalog of the 8 audit rules and their risk pattern signals, used by the audit tools.
Output schemas not formally documented. Findings structure (rule, severity, label, line, evidence) is inferred from _find() implementation, not declared in tool registration. LLMs cannot reliably parse or chain results without explicit schema.
No tool annotations despite all tools being read-only. FastMCP supports readOnlyHint, should be applied to all four tools to signal to LLMs that these are safe, non-destructive operations.
Parameter 'pattern' in audit_directory lacks constraints. Should document min/max length, valid glob syntax, and default behavior. Currently only has description 'File glob pattern to match (default: *.md)'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Error handling is minimal. _verdict() returns PASS/WARN/FAIL strings but does not guide LLM on recovery actions. E.g., if a file is not found, should return 'File not found at <path>. Check path and try again.' instead of bare error.
rule_list tool has empty input schema (no parameters). Description should clarify that this is a discovery/reference tool and when to call it (e.g., 'Call before auditing to understand the 8 patterns').