Python package for evaluating MCP (Model Context Protocol) server implementations using LLM-based scoring
The calculator_server.py example demonstrates solid foundational tool definition quality. All 6 tools have explicit names, descriptions, and properly typed input schemas. Naming follows verb_noun convention (add, subtract, multiply, divide, power, square_root). Descriptions are concise and action-oriented (10-50 chars), meeting the baseline for clarity. All parameters have type annotations (float) and descriptions. However, output schemas are not explicitly documented, the code implies numeric returns but lacks formal schema documentation in the tool definitions. Error handling is present in divide() and square_root() (ValueError for division by zero and negative square root), but error messages lack LLM-actionable recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are declared despite all tools being READ_ONLY and idempotent. The server is an evaluation/testing framework itself (pymcpevals), not a production tool server, which explains the minimal example scope.
Add two numbers together.
Divide first number by second number.
Multiply two numbers.
Raise base to the power of exponent.
Calculate square root of a number.
Subtract second number from first number.
Output schemas not documented. Tool definitions lack explicit return type schemas (e.g., {type: number, description: '...'}). LLMs cannot plan downstream operations or validate results without knowing the return structure.
Tool annotations missing. All 6 tools are READ_ONLY and idempotent, but these hints are not declared via readOnlyHint or idempotentHint. Agents cannot automatically determine tool safety properties.
Error messages lack recovery guidance. divide() and square_root() raise ValueError but do not guide the LLM on corrective action. Error message should be: 'Cannot divide by zero. Ensure denominator is non-zero.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 75 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 51 | - | v1 |
Parameter descriptions are minimal (under 30 chars). 'First number' and 'Second number' lack context on acceptable ranges or precision. Should specify: 'First number (float, any value)' or 'Base number (float > 0 for real exponents)'.