Evaluation suite for MCP AI agents — golden datasets, LLM-as-judge, security benchmarks
evalmcp exposes 4 evaluation tools with complete input schemas and descriptions. Naming follows verb_noun convention (list_suites, run_suite, evaluate, evaluate_rag). However, descriptions lack depth on WHEN to use each tool vs alternatives, parameter descriptions are minimal or absent, and output schemas are undocumented. The evaluate_rag tool exposes api_key as a parameter (security risk). No error handling guidance or recovery paths documented. Schemas are present but sparse on constraints and field documentation.
Judge a single output: compare actual_output against expected_output for a given input.
Score a RAG sample with native metrics: faithfulness (answer grounded in context), answer_relevancy, context_precision, context_recall (needs ground_truth). Uses an LLM judge — needs evalmcp[llm] + a model/api_key.
List the built-in benchmark suites (memory, security, reasoning, tool_use, humaneval, knowledge).
Run a built-in benchmark suite by name and return summary + metrics. Without a kernel pipeline this runs in self-test (dry-run) mode.
evaluate_rag exposes api_key as a required/optional parameter. Credentials must never appear in tool parameters, use server-side secret injection via environment variables or vault. Agent traces log all parameters, leaking secrets into logs and prompt history.
Output schemas are completely undocumented. LLMs cannot plan downstream calls or extract data without knowing what fields to expect. run_suite and evaluate return result payloads (input, tool, expected_output, actual_output, passed, score, latency_ms, judge_reasoning) but this is not declared in tool definitions.
Parameter descriptions are missing or minimal. 'judge' enum parameter lacks explanation of when to use 'exact' vs 'contains'. 'suite_name' description says 'One of the names from list_suites' but does not explain what each suite tests or when to choose one. 'metrics' array in evaluate_rag has no description of valid metric names.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. E.g., if run_suite fails because a suite name is invalid, the LLM has no recovery path.
evaluate_rag tool description mentions 'needs evalmcp[llm] + a model/api_key' but does not explain what happens if the optional dependency is missing, or how to configure the model. LLMs cannot infer fallback behavior.