MCP server for running security evaluations on LLMs, detecting data leakage, privilege escalation, and stateful leakage patterns. Provides REST API and tools for analyzing LLM security vulnerabilities.
Scoring was not performed
Missing output schemas across all 16 tools. LLMs cannot plan downstream chaining, extract specific fields, or validate responses against expected structure. Every tool should document its return type and key fields.
Enum constraints missing for constrained parameters. 'profile' should be enum [default, quick, custom]; 'provider' should be enum [auto, openai, anthropic, ollama, mock]. Free-form strings invite hallucinated values.
Numeric parameters lack bounds. 'scan_depth' (analyze_repository_security), 'offset'/'limit' (list_reports), 'limit' (get_trends) have no min/max. Unbounded integers invite absurd values (limit=999999).
Object parameters lack schema definition. 'tool_parameters' (test_mcp_tool_security), 'evaluation_results' (generate_security_report), 'custom_patterns' (redact) are all 'object' type with generic descriptions. Should define expected fields and types.
Tool naming lacks clear composition. 'test_data_leakage', 'test_redaction_effectiveness', 'test_mcp_tool_security' all start with 'test' but test different aspects with no clear prerequisites or sequencing. Unclear when to call which.
No error handling guidance. Tools provide no recovery hints (e.g., 'If repo_path not found, try list_available_repos()'; 'If report_id invalid, call list_reports() first'). Agents cannot self-correct.
Descriptions are generic or vague. Many lack explicit WHEN to use guidance. E.g., 'test_mcp_tool_security' doesn't explain whether it tests a tool in isolation or integrated into the evaluator.
Tool 'detect_privilege_escalation' has incomplete specification: no input schema visible, minimal description (~50 chars). Appears to be inferred rather than fully defined.
Async operation 'trigger_evaluation' lacks polling guidance. No mention of job_id, status field, or how to track completion. LLM cannot determine next steps.
Multiple redaction/detection tools ('redact', 'detect_sensitive_data', 'get_redaction_stats') lack clear composition guidance. Relationship between tools unexplained.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 0 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 48 | - | v1 |