MCP server for Patronus AI evaluation platform, providing tools for running evaluations, experiments, and managing evaluation criteria
Server has 6 tools with explicit schemas and descriptions, but quality is inconsistent. Tool names follow verb_noun convention (evaluate, run_experiment, batch_evaluate, list_evaluators, list_criteria, create_criteria). Descriptions are present but generic (10-100 chars, below the 194-char baseline). Parameter descriptions exist but lack actionable constraints, e.g., 'explain_strategy' enum is documented but no guidance on when to use each value. Output schemas are not documented; callers cannot predict response structure. Error handling returns generic {'status': 'error', 'message': str(e)} with no recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels (run_experiment and create_criteria are WRITE operations). Schemas are present and typed, but missing format constraints, ranges, and dependency documentation.
Run batch evaluations with multiple async evaluators
Create new evaluation criteria
Run a single evaluation using a remote evaluator with specified criteria and task data
List all available criteria from the Patronus API
List all available evaluators from the Patronus API
Run an experiment with multiple evaluators on a dataset
Output schemas not documented. Tools return raw Python dicts (e.g., {'status': 'success', 'result': result}) without specifying what 'result' contains. LLMs cannot plan downstream calls or extract structured data.
Tool descriptions are too short (40-50 chars) and lack actionable context. 'Run a single evaluation using a remote evaluator with specified criteria and task data' does not explain WHEN to use evaluate vs batch_evaluate, what the return value contains, or what happens on failure.
No tool annotations (readOnlyHint/destructiveHint). run_experiment and create_criteria are WRITE operations that modify state, but the schema does not declare this. Agents cannot distinguish safe (read) from risky (write) tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Error handling is generic and non-actionable. All exceptions return {'status': 'error', 'message': str(e)}. No recovery guidance, no error classification (retryable vs fatal), no invalid-value feedback. LLMs cannot self-correct.
Parameter constraints are incomplete. 'explain_strategy' is an enum but descriptions do not explain when to use 'never' vs 'on-fail' vs 'on-success' vs 'always'. 'max_concurrency' and 'limit' lack min/max bounds. 'api_key' is exposed as a parameter (security risk).