A comprehensive test automation framework combining Playwright + pytest with AI agents for test analysis, failure classification, and test generation. Includes a FastAPI results server with Allure reporting, Prometheus metrics, and mock e-commerce API.
This MCP server has significant definition quality gaps. Tool descriptions are present but generic (avg ~40 chars, well below the 194-char baseline). Input schemas exist but lack parameter descriptions, most parameters have no description field at all. The naming is reasonable (verb_noun pattern mostly followed), but parameter documentation is severely deficient. No output schemas are documented. Error handling is absent, tools return raw subprocess output or HTTP errors without recovery guidance. The server exposes internal implementation details (job IDs, file paths) rather than abstracting to user-friendly interfaces. No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels. Overall, this reads like an internal automation tool hastily wrapped as MCP, not a production-grade agent interface.
Check Allure report generation status
Get Allure report summary statistics
Get architecture documentation HTML
Create a shopping cart on external e-commerce site using test data
Search products on external e-commerce site (tutorialsninja.com/demo)
Generate Allure report from test results
Health check endpoint
Parameter descriptions missing or minimal. Most tools (run_pytest, read_job, read_config, etc.) have parameters with no description field. LLMs cannot infer what 'args', 'job_id', 'key', or 'results_dir' mean without explicit guidance.
No output schemas documented. Tools return raw subprocess output (stdout/stderr), file contents, or HTTP responses without specifying structure. LLMs cannot plan downstream calls or extract required fields.
No error handling or recovery guidance. Tools fail silently or return raw HTTP/subprocess errors. No indication of retryability, user-fixable vs fatal errors, or next steps for the LLM.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 32 | 2026-07-28+ | v2 |
Get Prometheus metrics
Read all test data from test_data.json
Read automation configuration from config.json
Get the status and output of a pytest job by ID
Read a specific test data key from test_data.json
Run pytest with specified arguments and return job status
Check if secrets file is configured
Tool descriptions are too short (avg 30 - 45 chars vs 194-char baseline). Descriptions like 'Get the status and output of a pytest job by ID' lack context on when to use this tool vs alternatives, what the output contains, or prerequisites.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools are marked with Risk levels (WRITE, READ_ONLY) in metadata but not exposed to the LLM via schema annotations. LLM cannot reason about side effects or retry safety.