MCP server for Mandoline's LLM evaluation tools
Mandoline MCP Server demonstrates solid definition quality with well-structured tools, clear naming conventions, and comprehensive parameter documentation. All 11 tools follow verb_noun naming patterns (create_metric, get_metric, update_metric, etc.). Descriptions are present and substantive (ranging 50-250 chars), explaining what each tool does and when to use it. Input schemas are complete with type definitions and parameter descriptions. However, there are notable gaps: (1) Output schemas are not explicitly documented, tools return responses but the response structure is not specified in the schema definition; (2) Error handling lacks recovery guidance, errors do not suggest next steps or categorize as retryable vs. fatal; (3) Tool descriptions could better specify prerequisites, side effects, and downstream dependencies; (4) Some parameter descriptions in the complex nested objects (prompt/response in create_evaluation and batch_create_evaluations) are vague about required structure. The batch tools (batch_create_metrics, batch_create_evaluations) show good composition patterns by accepting arrays, reducing token waste. Pagination is well-handled in list tools (get_metrics, get_evaluations) with skip/limit parameters and cost warnings in descriptions.
Evaluates the same pair against multiple metrics in one request.
Creates multiple metrics in one call.
Evaluates a single prompt/response pair and returns a score (-1 … +1).
Creates a new evaluation metric in Mandoline.
Retrieves one evaluation by ID.
Lists evaluations with pagination and filtering.
Output schemas not documented. Tools like create_metric, create_evaluation, and batch operations return responses but the response structure is not specified in tool definitions. LLMs cannot predict what fields are available in the response or plan downstream tool calls without this information.
Error handling lacks recovery guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. For example, update_metric on a non-existent metric_id will fail, but the tool description does not explain how to handle this or suggest alternatives.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 67 | <=2025-11-25 | v2 |
| 2026-03-09 | B | 70 | - | v1 |
Fetches a metric by its ID.
Lists metrics with filtering and pagination.
Check the server health status.
Updates evaluation metadata (score is immutable).
Modifies an existing metric's metadata.
Complex nested parameters (prompt and response objects in create_evaluation and batch_create_evaluations) lack clear documentation of required vs. optional nested fields and their structure. The description says 'Must be provided as an object' but does not specify what happens if invocations or context_assets are empty, or whether these are truly optional.
Tool descriptions do not explain side effects or idempotency. For example, create_metric does not state whether calling it twice with the same name+description creates a duplicate or returns the existing metric. This is critical for agent retry logic.
API Key is accepted as Authorization header (seen in server.ts validateAuthentication), but this is not documented in tool descriptions. LLMs may not understand that authentication is required and provided server-side, potentially leading to confusion about how to use these tools.