A continuous auditor for MCP servers that validates tools, schemas, and protocol compliance through deterministic testing and fixture-based validation
Three tools with minimal definitions. Schemas are empty objects with no properties defined. Descriptions are present but extremely terse (10-20 chars), below the 34-char p10 baseline. No error handling guidance, no output schema documentation, no parameter constraints. Tools appear to be read-only probes rather than substantive agent tools. This server is not suitable for agent integration.
Return how many datasets the Zürich CKAN catalogue currently lists.
Liveness probe — confirms the server is importable and serving tools.
Count records from the (mocked) upstream — drives the fixture-replay path.
All three tools have empty input schemas (no properties defined).
No parameter descriptions exist. Even though inputs are empty objects, the rubric requires every parameter to have a description explaining what it controls. This prevents LLMs from understanding parameter semantics.
Tool descriptions are 10-20 characters (health, dataset_count, record_count). Rubric baseline is 34-392 chars (p10-p90). These lack context for LLM selection.
No output schema documented. LLMs cannot plan downstream calls or extract data without knowing what fields to expect. Rubric requires 100% of A+ tools to have documented return types.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | 2026-07-28+ | v2 |
No error handling guidance. Tools provide no recovery hints, error classification, or actionable messages.