Evidence-driven protocol and agent-usability testing for MCP servers
Strong tool definitions with comprehensive schemas and clear descriptions. All 9 tools have explicit input schemas with typed parameters and descriptions. Tool annotations (readOnlyHint/destructiveHint) are properly declared. Naming follows verb_noun convention (register_target, list_targets, inspect_target, run_static_checks, measure_context_cost, generate_scenarios, run_target_tool). Descriptions range 50-180 chars, meeting the 10-1024 baseline. Key strength: security-conscious design (no secrets in params, explicit side-effect approval). Weakness: output schemas are not documented in the visible code, responses are cast to dict[str, Any] without schema definition, forcing LLMs to infer structure. Error handling guidance is minimal; tools do not return actionable recovery hints.
Return the supplied message unchanged for transport verification.
Generate task-oriented multi-tool journeys using sampling, with an explicit fallback.
Connect to a target and capture its complete paginated model-facing MCP contract.
List registered targets without resolving or exposing environment values.
Measure serialized model-facing tool metadata and exact duplicate descriptions.
Pretend to mutate state so approval behavior can be verified safely.
Register a secret-free stdio or Streamable HTTP target after boundary validation.
Output schemas not documented. All tools return dict[str, Any] without schema definition. LLMs cannot infer response structure, forcing them to guess which fields exist and their types. This violates the pattern:tool requirement that output schemas be documented.
Error handling lacks recovery guidance. Tools do not return actionable error messages. A failed target connection or invalid target_id returns a generic error without suggesting next steps (e.g., 'Target not found. Call list_targets() to see available targets.').
Pagination not implemented for list_targets. If many targets are registered, the response could grow unbounded. Should accept limit/offset parameters and return a total count.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Analyze live schemas, descriptions, annotations, and size without tool calls.
Execute one call; non-read-only tools require explicit side-effect approval.
generate_scenarios description mentions 'sampling' but sampling is deprecated in MCP 2026-07-28. Tool should clarify its fallback strategy when sampling is unavailable.
Test fixtures (echo, mutate) are exposed as production tools. These should be removed or gated behind a test mode flag to avoid confusion in production deployments.