Puppeteer Test Automation Framework with reporting and video recording
QA MCP is a test automation framework, not a production agent toolkit. Most tools accept Playwright Page/Locator objects as parameters, these are runtime objects that cannot be serialized or passed by an LLM. The schema definitions are present but fundamentally incompatible with how MCP tools must work: LLMs cannot instantiate Playwright Page objects, making 6 of 8 tools non-functional for agent use. Descriptions are present but often generic (e.g., 'Closes the student welcome modal by clicking the secondary button' lacks context on WHEN to call it or what prerequisites exist). No parameter enums, no output schema documentation, no error handling guidance. The server was clearly designed for Node.js test harnesses, not LLM agents.
Adds a textbook to a module within a course
Creates a new course with specified name, subject, dates, and plan. Returns a cleanup function to delete the course.
Creates a new module within a course with specified name. Returns a cleanup function to delete the module.
Closes the student welcome modal by clicking the secondary button
Clicks on 'OK, got it' buttons repeatedly until no more are found on the page
Generates Playwright test code from recorded actions
Runs a test with middleware that handles browser setup, login, test execution, and teardown
Sets up login function for a specific user role and environment
6 of 8 tools require Playwright Page/Locator objects as input parameters. These are runtime JavaScript objects that cannot be serialized, instantiated, or passed by an LLM. The schema declares type: 'object' with description 'Playwright Page object', but LLMs have no mechanism to create or provide such objects. This makes escapeUserGuide, escapeStudentWelcomeModal, createCourse, createModule, addTextbook, and runTest non-functional for agent use.
Output schemas are not documented. Tools like createCourse return a cleanup function (async () => { await deleteCourse(...) }), but there is no schema describing what the LLM receives. The agent cannot plan downstream operations without knowing the output structure. Pattern: tool (documented return types are required).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 28 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
No error handling documentation. Tools like createCourse interact with a web UI and can fail (timeouts, missing buttons, page navigation errors). There is no guidance on what errors can occur, whether they are retryable, or how to recover. Example: if the 'Create new course' button is not found, the tool throws an error with no recovery path documented.
Descriptions lack actionable context. Examples: 'Closes the student welcome modal by clicking the secondary button' (35 chars) does not explain WHEN to call it (post-login? first load?) or what happens if the modal is not present. 'Runs a test with middleware...' does not clarify what the middleware does or what prerequisites exist (logged-in user? test data setup?).
Parameter descriptions are vague or missing type hints. Example: createCourse accepts options.startDay and options.endDay as strings with no format constraints. Are these ISO dates? Day numbers? Month/day strings? The description 'Start day for the course' does not specify format. LLMs will guess and pass invalid values.
No enums for constrained choices. Example: runTest accepts setupInstructorLogin, setupStudentLogin, setupSuperAdminLogin, setupCollegeInstructorLogin. The valid roles are hardcoded in the parameter names, not enumerated. An LLM may invent roles like 'setupTeacherLogin' and pass invalid configuration.
Schemas use bare 'object' types without property definitions. Example: createCourse parameter 'ctxt' is declared type: 'object' with description 'TestContext object', but no properties are listed. An LLM cannot determine what fields ctxt must have or what values to pass.