An AI agent security and compliance monitoring system that detects prompt injection attacks, enforces role-based access control, and tracks risk scores for tool execution with OpenTelemetry observability.
Sentinel Trace has 4 tools with basic descriptions and visible input schemas, but significant quality gaps prevent production use. Tool names follow verb_noun convention (search_policy, retrieve_evidence, update_record, delete_record), which is good. However, descriptions are minimal (10-60 chars), parameter descriptions lack detail, and critical security/error handling patterns are absent. No output schemas documented. The retrieve_evidence tool accepts an attack_mode parameter that simulates poisoned data, a red flag for security. Mock implementations and lack of real error handling further limit readiness.
Mock deletion of a record.
Retrieve evidence from fixtures, simulating an attack if requested.
Mock search for compliance policies.
Mock update of a record.
Descriptions are too short (10-60 chars) to guide LLM tool selection. Baseline is 194 chars. Missing WHEN to use each tool and what it returns.
No output schemas documented. LLMs cannot plan downstream calls or extract required fields. Baseline: 100% of A+ tools document return types.
retrieve_evidence accepts attack_mode parameter to load poisoned data. This is a security anti-pattern, tools should never accept flags to bypass safety or inject malicious content.
No error handling guidance. FileNotFoundError returns '{}' with no recovery hint. LLMs need actionable error messages: 'Fixture not found. Check fixture path or call search_policy() first.'
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | <=2025-11-25 | v2 |
Destructive tools (delete_record, update_record) lack confirmation or dry-run support. No permission checks visible. Agents can delete records without safeguards.