MCP server for RLHF human preference annotation, rubric evaluation, and red teaming vulnerability assessment with DPO dataset generation
Critical gaps across all dimensions. Tool 1 (annotate_rlhf_preference_pair) has a reasonable description (168 chars) and three parameters with types and descriptions, meeting baseline minimums. Tool 2 (execute_skill_action) has NO visible schema, parameters, or description in the source code, only a name and risk classification. Neither tool has documented output schemas. Parameter descriptions lack constraints (enums, ranges, formats). No error handling guidance. No security considerations (e.g., how are rubric criteria validated?). The server is STDIO-only, which is a hard transport cap at 50. Overall definition quality is severely limited by missing schemas and incomplete parameter documentation.
Annotate RLHF preference pairs using rubric criteria with inter-annotator agreement scoring, domain expert verification, and red teaming vulnerability assessment
Execute skill action for the RLHF human preference rubric red teaming pipeline
Tool 'execute_skill_action' has NO description, parameters, or input schema visible in source code. Only name and risk classification present. Cannot determine when/why to call it or what it accepts.
No output schemas documented for either tool. LLMs cannot plan downstream calls or extract required fields (e.g., annotation_batch_id, preferred_completion). Violates pattern:tool baseline (100% of A+ tools have documented return types).
Parameter 'rubric_criteria' accepts free-form strings with no enum constraint. Description lists examples ('FACTUALITY_AND_CONCISENESS', 'CODE_SAFETY_AND_ROBUSTNESS') but no formal constraint. LLMs will hallucinate invalid criteria values.
No error handling guidance. What happens if rubric_criteria is invalid? If model completions are too long? If inter-annotator agreement is below threshold? LLM receives no recovery path.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | 2026-07-28+ | v2 |
Tool naming: 'execute_skill_action' is generic and vague. Does not follow verb_noun pattern. 'execute' and 'action' are too abstract, LLM cannot infer intent from name alone. Should be specific (e.g., 'run_red_teaming_pipeline', 'generate_dpo_dataset').