Conformance test kit for MCP servers — schema, encoding, and path-safety checks seeded by real upstream bugs
mcp-audit is a conformance test kit, not a production MCP server. Its 8 tools are deliberately buggy fixtures designed to validate auditing logic, not to provide real functionality. Tool descriptions are adequate for test purposes but lack LLM-optimization. Schemas are present and mostly complete, but parameter descriptions are minimal or absent. The echo_server.py fixture is correct; fixture_server.py intentionally violates patterns (loose_required, hidden_beta, mojibake_read, drive_letter_create) to seed test cases. This is appropriate for a test harness but scores poorly against production tool standards.
Creates files; silently accepts C:\ style paths on POSIX.
Echo the provided text back.
Runtime requires 'beta' which is NOT in advertised required.
Advertises a required field but never enforces it.
Reads a file chunk boundary; corrupts straddling UTF-8.
First N bytes of a file; decodes each 1024-byte chunk independently (corrupts straddling UTF-8).
First N characters of a file; decode-then-slice (boundary-safe).
Test fixtures intentionally violate schema/runtime contracts (loose_required, hidden_beta). While appropriate for a test kit, these tools would fail production audits and should not be deployed as real MCP servers.
Parameter descriptions are minimal or absent. 'path' and 'head' in read_head/read_head_safe lack guidance on format, constraints, or expected values. LLMs cannot infer parameter semantics from names alone.
No output schemas documented. Tools return CallToolResult with TextContent, but the structure and semantics of returned text are not specified. LLMs cannot plan downstream actions without knowing output shape.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Echoes text; validates exactly what it advertises.
Error handling is minimal. Tools return is_error=True with text messages, but do not categorize errors as retryable/user-fixable/fatal or provide recovery guidance. Agents cannot reason about next steps.
drive_letter_create is annotated read_only_hint=false but lacks destructiveHint or idempotentHint. Tool annotations are incomplete for destructive operations.