MCP Server for computing DPO loss and implicit rewards for pairwise preference pairs using Direct Preference Optimization
Two tools with complete JSON schemas and basic descriptions, but significant gaps in parameter documentation and output schema clarity. Tool names lack action verbs (evaluate_* is weak). Parameter descriptions are minimal or absent. No error handling guidance. Output schemas are not documented, LLMs cannot predict response structure. No tool annotations (readOnlyHint, etc.). Batch tool has severely underdocumented nested array structure.
Evaluate batch of pairwise samples
Compute DPO loss and implicit rewards for a chosen vs rejected pair
Tool names lack action verbs. 'evaluate_preference_pair' and 'evaluate_batch' are weak; should be 'compute_dpo_loss' or 'calculate_implicit_reward' to clarify the mathematical operation performed.
Parameter descriptions are missing or trivial. 'pi_w', 'ref_w', 'pi_l', 'ref_l' have no descriptions in the schema, LLMs cannot infer that these are log probabilities. 'beta' description is present but minimal. Violates pattern:tool-description.
Output schema is not documented. Tool descriptions do not specify what fields the response contains (e.g., 'reward_chosen', 'dpo_loss', 'accuracy'). LLMs cannot plan downstream operations without knowing response structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
evaluate_batch 'pairs' parameter has no item schema. Nested object properties (pi_w, ref_w, pi_l, ref_l) are not defined in the inputSchema, only 'type: object' with no properties. LLMs cannot validate structure before calling.
No error handling or recovery guidance. If an LLM passes invalid log probabilities (e.g., positive values when negatives expected), the tool will silently compute wrong results. No validation, no error messages, no guidance on valid ranges.