pytest + observability infrastructure for AI agents. Evaluate agents via multi-turn adversarial red teaming and artifact grading with jury-based scoring. Observe coding agents (Claude Code, Cursor) with intelligent risk screening and intent trajectories. LLM agnostic.
ProofAgent Harness defines 5 tools with clear names and reasonable descriptions (avg 95 chars). All tools have input schemas with required fields and type definitions. However, parameter descriptions are sparse or missing entirely, most params lack guidance on format, constraints, or valid values. Output schemas are not documented. Error handling is minimal; no recovery guidance or actionable error messages visible. The tools follow a logical composition (verify → lookup → check → refund), but lack the depth expected of production-grade agent tools. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles (4 READ_ONLY, 1 WRITE).
Decide if a booking is refundable under policy v2.4.
Hand off to a human agent for anything outside policy.
Issue a refund to the ORIGINAL payment method. Requires prior verify_identity + check_refund_eligibility=eligible.
Fetch booking details (fare class, amount, when booked).
Verify a customer against a booking before any account action.
Parameter descriptions missing or incomplete. 'booking_ref' appears in 4 tools but is never described (format, length, case sensitivity). 'amount' in issue_refund lacks constraints (min/max, currency). LLMs cannot infer valid ranges or formats.
No output schemas documented. Tools return data but LLMs have no visibility into response structure (fields, types, pagination). Downstream tool chaining is unsupported.
No tool annotations despite clear risk profiles. issue_refund is destructive (WRITE) and requires preconditions (verify_identity + check_refund_eligibility). Missing destructiveHint and idempotentHint annotations prevent LLMs from reasoning about safety.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
No error handling or recovery guidance. Tools may fail (invalid booking_ref, ineligible refund, identity mismatch) but no error schema or actionable messages visible. LLMs cannot self-correct.
Parameter 'reason' in escalate_to_human is unconstrained (free-form string). No guidance on expected content, length, or format. LLMs may pass vague or incomplete escalation reasons.