EU AI Act trace compliance agent. Evaluates production LLM traces against EU AI Act (Regulation 2024/1689) criteria using Gemini 3, Google Cloud Agent Builder (Vertex AI Agent Engine), and Arize Phoenix MCP for trace collection and annotation.
AuditLens exposes 3 tools with significant definition gaps. Tool names lack action verbs (get-spans, add-dataset-examples, list-traces use hyphens inconsistently). Descriptions exist but are vague and lack WHEN/WHY context. Input schemas are present but incomplete: parameters lack type constraints, enums, or validation guidance. No output schemas documented. Error handling is absent, no recovery guidance or categorization. The tools appear to delegate to external MCP servers (Phoenix) rather than implementing core logic, which limits their autonomy and clarity.
WRITE: append failing examples to a violations dataset via `add-dataset-examples`. Adds trace examples with input, output, and compliance scoring metadata to a Phoenix dataset.
READ: raw spans for the project via the partner MCP `get-spans` tool. Returns the spans array in the same shape src/phoenix.js already collapses.
List recent traces from Phoenix project for the dashboard. Fetches up to 25 traces for compliance evaluation.
Tool names use hyphens (get-spans, add-dataset-examples) instead of snake_case verb_noun convention. LLMs expect get_spans, add_dataset_examples. Inconsistent naming increases parsing errors.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. get-spans should document the 'spans array' structure; add-dataset-examples should confirm success/failure per example; list-traces should document trace object fields.
Descriptions lack WHEN/WHY context and are under 100 chars. 'READ: raw spans...' and 'WRITE: append failing examples...' state WHAT but not WHEN to call or what prerequisites exist. Baseline: 194 chars average for A+ tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Input parameters lack validation constraints. 'limit' is a number with no min/max (baseline: 1 - 100 for pagination). 'project_identifier' is a string with no format or length guidance. 'examples' array has no item schema or count limits.
No error handling or recovery guidance. If get-spans fails (invalid project, rate limit, timeout), the LLM has no guidance on retry strategy, fallback, or user-fixable actions. Baseline: A+ tools categorize errors as retryable/user-fixable/fatal.