Loading ToolBench
Multi-package security framework for AI agents covering policy enforcement (WARD.md), harness adapters (Antigravity, Claude Code, MSAF), MCP security proxy, vault/encryption, supply chain scanning, compliance, and observability
Weave Protocol exhibits severe definition quality gaps across all 16 tools. While tool names follow verb_noun convention (scan_package, detect_prompt_injection, encrypt_capability), descriptions are present but lack LLM-optimization depth. Critical issue: input schemas are visible in the metadata but parameter descriptions are minimal (1-2 sentences). Output schemas are entirely undocumented, no tool specifies what fields are returned or their types. Error handling is absent; no tool provides recovery guidance or categorizes errors as retryable/fatal. The security-focused domain (supply chain scanning, policy enforcement, adversarial testing) demands precise parameter constraints and clear error paths, but these are missing. Example: scan_package accepts 'registry' as enum [npm, pypi] but lacks guidance on what happens if a package is not found, what fields are returned, or how to chain results to downstream tools. run_attack_suite and run_agentsecbench are marked DESTRUCTIVE but have no confirmation/dry-run pattern documented. Composition is weak: encrypt_capability and decrypt_capability exist but no tool documents the vault structure, key rotation, or recovery procedures.
Decrypt and retrieve agent capabilities from cryptographic vaults
Detect behavioral drift in agent tool usage patterns
Detect indirect prompt injection (IPI) attacks in browser agent inputs and web content
Detect exposed secrets, API keys, and credentials in content
Encrypt and contain agent capabilities using cryptographic vaults
Enforce WARD.md security policies on agent tool calls and capabilities
Intercept and gate MCP tool calls in real-time with security scanning
Real-time monitoring and logging of agent activity with compliance tracking
Output schemas completely undocumented. No tool specifies return type, field names, or structure. LLMs cannot plan downstream tool calls or extract required data.
Destructive tools (run_attack_suite, run_agentsecbench) lack confirmation/dry-run pattern. No mechanism to prevent accidental execution against production agents.
Error handling absent across all tools. No guidance on retryability, user-fixable errors, or recovery paths. Errors like 'package not found' or 'vault access denied' provide no actionable next steps.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | <=2025-11-25 | v2 |
Run standardized, versioned, citable AI agent security benchmark with curated attack suites
Execute adversarial attack suite against agent (68+ documented attacks across 5 categories)
Scan Docker images for tag overwriting, layer vulnerabilities, and supply chain risks
Scan VS Code, Cursor, and JetBrains IDE extensions for malicious code and supply chain attacks
Scan npm/PyPI packages for typosquats, CVEs, sandwich-pattern attacks, and malicious dependencies
Scan content for personally identifiable information (PII)
Verify agent intent and authenticity using cryptographic proofs
Verify thread identity and intent using blockchain anchoring (Solana/Ethereum)
Parameter descriptions are minimal (1-2 sentences). Missing format constraints, ranges, and dependencies. E.g., 'wardPolicy' parameter lacks schema structure; 'attackCategory' enum is present but no guidance on which category to use for which threat model.
Cryptographic tools (encrypt_capability, decrypt_capability, verify_thread_identity) lack documentation of key management, rotation, recovery, and blockchain integration details. No guidance on vault structure or failure modes.