AI-Native Quality Engineering Lab with MCP, automated testing and intelligent QA agents
Server has 6 tools with basic schemas and descriptions, but significant gaps limit production readiness. All tools have descriptions (10-100 chars, mostly adequate) and input schemas with type definitions. However, parameter descriptions are sparse or missing context. Error handling is present but generic. No tool annotations (readOnlyHint/destructiveHint). Output schemas are undocumented, LLMs cannot predict response structure. Tool naming is clear (verb_noun pattern), but composition issues exist: run_api_tests has no input parameters despite being a WRITE operation, creating ambiguity about what it executes. Missing pagination, result limits, and chaining IDs in responses. No evidence of idempotency guarantees or recovery guidance in errors.
Analyzes a test failure using an LLM and returns probable root cause, category, evidence, recommended action and confidence.
Evaluates automated test metrics against Quality Engineering quality gates.
Generates risk-based QA test scenarios in BDD/Gherkin format using an LLM.
Returns the latest real quality status based on Playwright execution results.
Runs the Playwright API automated test suite and returns real execution metrics.
Analyzes a failed automated test and proposes a safe remediation without modifying code automatically.
run_api_tests has empty input schema (no parameters) but is marked WRITE risk. LLM cannot determine what tests run, what environment, or what configuration applies. Ambiguous tool purpose.
No output schemas documented for any tool. LLMs cannot predict response structure, field names, or types. Forces trial-and-error parsing and wastes tokens on unstructured text responses.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). run_api_tests marked WRITE but lacks destructiveHint. LLM cannot reason about retry safety or side effects.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Error responses return generic JSON with tool name and message. No recovery guidance (e.g., 'Try X next'), no error classification (retryable vs fatal), no invalid value echoed back for self-correction.
get_quality_status returns verbose Playwright JSON report fields (stats, source, generatedAt). No pagination or result limits documented. Response bloat wastes tokens and risks context window exhaustion.