MCP server for evaluating AI agent defenses against jailbreak and red team attacks
This server has severe structural and definition quality issues. Tool definitions are minimal, descriptions lack actionable detail, input schemas are either missing or severely incomplete, and the overall architecture suggests an incomplete proof-of-concept rather than a production-ready MCP server. The mcp_server.py file does not implement the MCP protocol correctly, it merely prints a JSON status message to stdout once and exits, with no actual tool registration, request/response handling, or capability negotiation. The evaluate_defenses tool in client.py is not properly integrated into the MCP server registration. Tool names use generic verbs but lack specificity in describing what action is taken. Descriptions are present but too brief and lack guidance on when to call each tool, what parameters mean, or what the output contains.
Evaluate defenses against jailbreak prompts by analyzing a list of prompts and calculating the defense success ratio
Execute a skill action in the agent playground for red team evaluation
No input schemas defined for execute_skill_action. The tool is registered with only a description; no parameters, types, or constraints are visible in the source code.
Input schema for evaluate_defenses is minimal and lacks proper JSON Schema structure. The 'jailbreak_prompts' parameter is declared as a list but with no minItems, maxItems, validation rules, or description of what makes a valid jailbreak prompt. Parameters lack descriptions explaining their role or constraints.
Tool descriptions are under 100 characters and lack actionable context. 'Execute a skill action in the agent playground for red team evaluation' does not explain what skill actions are, what parameters are needed, what the return value contains, when to use this tool instead of another, or what side effects occur. This violates the pattern requirement that descriptions must answer WHAT, WHEN, and WHAT IT RETURNS.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 22 | 2026-07-28+ | v2 |
Output schemas are completely undocumented. For evaluate_defenses, the code returns {'defended_ratio': ..., 'vulnerability_details': ...} but the MCP definition does not declare this structure, field types, or what each field means. LLMs cannot plan downstream operations or extract data without knowing the output schema. This violates the pattern requirement to document output schemas.
The evaluate_defenses implementation uses a simplistic heuristic (checking if 'ignore' appears in prompts) to classify jailbreak success. No error handling, no guidance on what 'defended_ratio' means, no explanation of the 'vulnerability_details' format, and no indication of what values are possible. LLMs cannot reason about this tool's reliability or when to retry.
No per-parameter descriptions for jailbreak_prompts in evaluate_defenses. The parameter name alone does not clarify what format these prompts should be in, what constitutes a valid jailbreak attempt, or what the tool does with them. LLMs cannot reliably construct valid inputs.
Tool naming is generic. 'execute_skill_action' is vague, what is a 'skill action', what does 'execute' do, and what parameters control this execution? The name does not convey the intent clearly enough for LLMs to decide when to call it. Should be more specific, e.g., 'run_red_team_scenario' or 'test_agent_against_jailbreak'.
No error handling or guidance. Neither tool declares what errors are possible, what they mean, or what the LLM should do in response. If evaluate_defenses receives invalid prompts, there is no error message to guide recovery. This violates the pattern requirement that error responses must tell the LLM what to do next.
No tool composition or chaining hints. If execute_skill_action returns results, there is no documentation of what IDs or references those results contain, so the LLM cannot chain to evaluate_defenses or other tools. Broken chains force discovery detours.