Loading ToolBench
Security guardrails for AI agent payments. Autonomous security guardrails with red team attack simulation and blue team defense for adversarial co-evolution.
Three tools with complete input schemas and descriptions, but significant gaps in output documentation and error handling. Tool names follow verb_noun pattern (sentinel_audit_transaction, sentinel_revoke_token, sentinel_check_nonce). Descriptions are detailed (100-200 chars) and explain security purpose. Input schemas are well-typed with descriptions for all parameters. However, output schemas are not formally documented, only inferred from code. Error handling is minimal; no recovery guidance or actionable error messages. No tool annotations (readOnlyHint/destructiveHint). Missing pagination, rate limiting, and per-item error reporting for batch operations.
Run a full 9-check security audit on an agent payment transaction. Checks: token integrity, budget enforcement, merchant allowlist, category restriction, expiry, rate limit, amount mismatch, replay attack, and revocation evasion. Returns PASS/BLOCKED with per-check detail and a tamper-evident SHA-256 audit hash.
Check if a nonce has been used before (replay detection). Returns whether the nonce is fresh or has been seen.
Revoke a token by ID, preventing any further use. Persists revocation state to ~/.sentinel/revoked_tokens.json.
Output schemas not documented. Tool descriptions explain what checks run but do not specify the structure of returned audit results, check objects, or error responses. LLMs cannot plan downstream operations without knowing what fields to extract.
No error handling guidance. Code returns raw exceptions (e.g., 'error': str(e) in check_expiry). LLMs receive unstructured error text with no recovery hints. Should return categorized errors: 'Invalid ISO 8601 timestamp. Expected format: 2024-01-15T10:30:00Z'.
No tool annotations. sentinel_revoke_token is destructive (modifies ~/.sentinel/revoked_tokens.json) but lacks destructiveHint. sentinel_audit_transaction is read-only but lacks readOnlyHint. Annotations help LLMs reason about side effects and retry safety.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Persistent state files (~/.sentinel/used_nonces.json, ~/.sentinel/revoked_tokens.json) are not documented as tool side effects. LLMs do not know that sentinel_check_nonce and sentinel_revoke_token modify local state. Should document in descriptions: 'Persists nonce to ~/.sentinel/used_nonces.json for replay detection.'
Parameter 'approved_amount' is optional but its absence changes behavior (defaults to transaction amount). Undocumented dependency: if omitted, amount_mismatch check always passes. Should state: 'If omitted, defaults to amount (no mismatch check).'