A comprehensive red-teaming and QA testing framework for AI agent plugins across three companies (KakaoPay Securities, Musinsa, SamilPwC). Includes automated multi-turn adversarial testing, LLM-based synthetic data generation, transcript analysis, and guardrail validation.
This server exhibits severe definition quality gaps across all dimensions. Tool descriptions are present but generic (avg ~80 chars, well below the 194-char baseline). Input schemas are visible but lack proper type definitions for complex parameters (e.g., 'session' in run_full_session is typed as 'object' with no nested schema). Parameter descriptions are minimal and often omit constraints, ranges, or format guidance. Output schemas are entirely undocumented, no tool specifies what fields it returns or their types. Error handling is absent; tools provide no recovery guidance. The server appears designed for internal hackathon testing rather than production agent integration. No tool demonstrates idempotency guarantees, permission gates, or audit trail support.
Parses JSONL transcript logs to extract tool usage statistics, including tool call distribution, file modifications, and command patterns across multiple submissions.
Uses Gemini LLM to enrich fashion product data with MD-level analysis including TPO tags, fit coverage, return risk notes, and inventory status.
Locates the codex CLI binary from standard installation paths or system PATH, with fallback error handling.
Uses Gemini LLM to generate synthetic peer group investment data for KakaoPay Securities testing, including hold ratios, loss-cut percentages, and FOMO trigger keywords.
Uses Gemini LLM to generate synthetic corporate SOP (Standard Operating Procedure) compliance rules for SamilPwC testing, including penalty levels and rule text.
Analyzes plugin response for immediate guardrail violations specific to each company (forbidden keywords, PII leakage, role hijacking, disclaimer omission).
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. Violates pattern:tool and pattern:response-shaper.
Complex parameters lack proper schema definition. 'session' in run_full_session is typed as 'object' with no nested properties documented. Violates pattern:constrained-input.
No error handling or recovery guidance. Tools provide no actionable error messages, retryability classification, or next-step hints. Violates pattern:recovery-guide.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 42 | <=2025-11-25 | v2 |
Executes a complete multi-turn red-teaming session (10-20 turns) against a plugin, automatically resuming conversation and flagging guardrail violations.
Executes a single turn of codex exec or codex exec resume, running an adversarial prompt against a plugin and capturing the response. Returns session_id, response text, and success status.
Stop-hook helper that saves AI chat transcripts into the submission's logs/ folder, with optional slimming to conversation-only lines.
Scrapes Musinsa fashion e-commerce website for base product data (titles, prices, brands) with fallback procedural generation if scraping fails.
Filters a raw transcript (JSONL) to keep only conversation lines (user/assistant messages), dropping tool calls, thinking blocks, and metadata. Returns slimmed JSONL or None for fallback.
Descriptions are generic and lack WHEN/WHY context. E.g., 'Executes a single turn...' does not explain when to use run_single_turn vs run_full_session. Violates pattern:tool-description.
No pagination, limits, or result-capping documented. scrape_musinsa_base_data accepts 'limit' but no max is specified; no total count or next_cursor returned. Violates pattern:paginated-result.