REST API and MCP server for benchmark regression analysis using AI-powered comparison of performance test runs
Server has 4 tools with basic schemas and descriptions, but significant gaps limit production readiness. Tool names lack clear action verbs (compare, ask, filters, health are weak). Descriptions are present but generic (10-50 chars for most). Input schemas exist but lack parameter descriptions, critical for LLM reasoning. No output schemas documented. Error handling absent. Parameters like 'detail_level' and 'field' lack context on valid values beyond enums. The 'ask' tool re-runs comparison before answering, which is inefficient and not documented as a side effect.
Ask a follow-up question about a comparison. Re-runs the comparison first so the AI has full context.
Compare two benchmark runs and get AI regression analysis
Get available dropdown values for a filter field (cloud, os_vendor, etc.)
Health check endpoint
Tool names lack action verbs. 'compare', 'ask', 'filters', 'health' do not follow verb_noun pattern (e.g., get_filters, run_comparison). LLMs cannot infer intent from these names alone.
No parameter descriptions in schemas. 'run1', 'run2', 'detail_level', 'field', 'filters', 'question', 'chat_history' lack explanations of what they control or how to populate them. LLMs cannot reason about parameter selection.
No output schemas documented. Tools return data but LLMs have no visibility into response structure, field names, or types. Prevents downstream tool chaining and forces LLMs to guess field names.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
'ask' tool re-runs comparison before answering (per description). This side effect is not documented in the schema and wastes compute. Should be explicit in description or split into separate tools.
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. A failed comparison returns nothing actionable.