A tool for running evaluations against an MCP server using the Anthropic API
This is a test server for the mcp-evals CLI tool, not a production MCP server. All 6 tools have clear verb-prefixed names (add, echo, get_*) and valid JSON schemas with typed parameters. Descriptions are present but terse (8-56 chars), falling short of the 50-200 char guidance for LLM-optimized descriptions. All tools are READ_ONLY and low-risk. Schemas are well-formed with proper type definitions and parameter descriptions. However, no output schemas are documented, no error handling guidance exists, and descriptions lack context about WHEN to use each tool or what dependencies exist between them. This is acceptable for a reference test server but would not meet production standards.
adds two numbers together
echoes back the input message
returns the current time
retrieves an environment variable value
retrieves system logs for a service (simulates log aggregation system)
retrieves user information (simulates Buildkite-style API)
Output schemas not documented. No tools declare return types, fields, or structure. LLMs cannot plan downstream actions or extract needed data.
Descriptions are too terse (8-56 chars, baseline 50-200). 'returns the current time' and 'echoes back the input message' lack context about WHEN to call the tool or what the LLM should do with the result.
No error handling or recovery guidance. Tools do not document what happens on failure (e.g., env var not set, service unavailable, invalid user_id) or what the LLM should do next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 65 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
get_env tool accepts any environment variable name without validation. No guidance on which env vars are safe to expose. No documentation of which vars are available, defaults, or constraints.
get_system_logs optional parameters (level, lines) lack value constraints. 'log level filter (info, warn, error)' should be an enum; 'lines' should specify min/max bounds (e.g., 1-1000).