Test whether your AI agent respects permission boundaries before production access. A policy engine that evaluates permissions against synthetic user policies and logs all decisions to an audit trail.
Agent Guardrail exposes 2 tools via FastAPI HTTP. Both tools have descriptions and basic input schemas, but critical gaps prevent higher scores. Tool naming is weak ('check_and_simulate' violates verb_noun clarity; 'healthz' is non-standard). Schemas lack proper type definitions for several parameters. Descriptions are present but generic (check_and_simulate: 67 chars, healthz: 95 chars, below the 194-char baseline). No output schemas documented. Error handling is minimal, HTTP status codes (200/403/404) are returned but no actionable recovery guidance for LLMs. No tool annotations (readOnlyHint, destructiveHint). The audit.py module shows logging infrastructure but no per-request logLevel support or structured error responses.
Evaluate permission for an action on a resource for a user against the policy. Returns 200 if allowed, 403 if denied, 404 if user not found.
Health check endpoint that validates the policy configuration is valid and loadable.
Tool naming violates verb_noun convention. 'check_and_simulate' contains 'and' (multiple responsibilities) and is vague about what 'simulate' means. 'healthz' is non-standard (should be 'check_health' or 'get_health_status'). LLMs cannot infer intent from these names alone.
No output schemas documented. LLMs cannot plan downstream calls or extract required fields (e.g., does check_and_simulate return allowed:bool, detail:str, or HTTP status only?). Baseline: 100% of A+ tools document return types.
Error handling lacks recovery guidance. HTTP 403/404 responses provide no actionable next steps for LLMs. Pattern requires: 'User not found. Try search_users() with a partial name.' Here, a 404 should explain what to do (e.g., 'User not found, verify user_id format or check available users').
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
Parameter descriptions are minimal. 'resource' (67 chars total for tool description) lacks format/enum constraints. Should specify: 'One of: emails, records, calendar, documents' or a regex pattern. 'action' similarly lacks enum (read, write, delete, etc.).
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Both tools are READ_ONLY per metadata, but this is not declared in the tool schema. LLMs cannot infer safety properties without explicit annotations.