A benchmark corpus of MCP servers with security test cases covering authorization, command injection, path traversal, SSRF, and tool poisoning vulnerabilities
This is a benchmark corpus, not a production MCP server. The 16 tools are duplicates across multiple test cases (authz-001 through authz-004, authz-002, authz-003). Tool definitions are present with schemas and descriptions, but the corpus is designed to test security vulnerabilities (authorization bypass, confused deputy, privilege escalation), not to demonstrate quality tool design. Descriptions are minimal (10-50 chars), parameters lack detailed constraints, and output schemas are not documented. No error handling guidance. The server is intentionally vulnerable in some cases (authz-001, authz-002, authz-003 non-safe variants) to test detection. This is not a production-grade tool suite.
Fetch an expense report.
Fetch an expense report.
Fetch an expense report.
Fetch an expense report.
List open support tickets for a tenant.
List open support tickets for a tenant.
Read a configuration setting.
Read a configuration setting.
Duplicate tool definitions across test cases. The 16 tools listed are the same 4 tools repeated 4 times from different corpus cases (authz-001, authz-002, authz-003, authz-004). This inflates the tool count and suggests the corpus is a test harness, not a cohesive server.
Descriptions are too short (10-50 chars). Rubric baseline is 194 chars average; these are 30-50 chars. 'Fetch an expense report' and 'Report the authenticated user' lack context on when to use, what data is returned, or prerequisites. LLMs cannot reliably select these tools.
Output schemas not documented. Tools return strings (e.g. 'no report for X in Y', 'not authorized to read reports for X'). LLMs cannot plan downstream calls or extract structured data. No field definitions, no pagination, no error structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Report the authenticated user for this session.
Report the authenticated user for this session.
Report the tenant this session is authenticated as.
Report the tenant this session is authenticated as.
Report the role this session is authenticated as.
Report the authenticated user for this session.
Report the authenticated user for this session.
Report the role this session is authenticated as.
Parameter descriptions lack constraints. 'user' param in get_report has description 'whose report to fetch' but no guidance on format (email, username, ID?). 'role' param in read_setting has description 'the caller's role' but no enum of valid roles (admin, viewer, owner?).
No error handling guidance. Tools return plain strings like 'not authorized to read reports for X' or 'no report for X in Y'. LLMs cannot distinguish retryable errors from user-fixable ones. No recovery suggestions.