Loading ToolBench
MCP-driven QA workbench: drive web/API/DB through MCP, emit durable, human-editable test artifacts (JSON canonical → Playwright TS + Gherkin), with cross-session agent memory.
Proba MCP has 17 tools with inconsistent quality. Most tools have descriptions (good), but many lack complete input schemas. Parameter descriptions are present but often vague (e.g., 'Locator (optional)' in proba_act lacks format/strategy details). Output schemas are not documented anywhere. Error handling is absent, no recovery guidance, no categorization. Naming is verb-forward (good), but composition issues exist: proba_act combines navigate/click/fill/select/check/wait/expect into one tool, violating single-responsibility. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels. Baseline: 549 production tools average 18-char names and 194-char descriptions; Proba averages ~20 chars (good) but descriptions average ~110 chars (below baseline), and many lack actionable detail.
Store a named test account for an app (idempotent on name). Reference its fields in any step value as {{account.<name>.<field>}} — e.g. {{account.client.email}} / {{account.client.password}}. Keeps credentials out of the test artifacts and lets one flow run against different accounts/roles.
Execute and record a web action (navigate/click/fill/select/check/wait/expect). Locators must be role/text/testid — positional css is rejected.
Create an app (surface: web/mobile/admin…) under a project. Its key becomes the appKey for sessions and scope.
Remove a test account or variable from an app.
List an app's test accounts and variables (secret values are masked). Use to see what {{account.*}} / {{var.*}} references are available before writing steps.
Store a named variable for an app (idempotent on name). Reference it in any step value as {{var.<name>}} — e.g. a base URL, a coupon code, an environment value.
Three tools (proba_diagnose, proba_patch_step, proba_remember) have empty input schemas ({}). These tools are unusable without parameter documentation.
proba_act combines 7 distinct actions (navigate/click/fill/select/check/wait/expect) into one tool. Violates single-responsibility pattern. LLM must reason about which action to invoke; composition is brittle. Should split into separate tools or use a more granular action enum with conditional parameters.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. proba_project_list should document returned fields (projectKey, name, apps array, etc.). proba_session_open should document what 'known selectors' structure looks like.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
Live candidate locators for a failed step to help fix it.
Emit the recorded test case as Gherkin + JSON + Playwright artifacts.
Fix a failed step by updating its locator or action.
Create a project (a client/workspace). Returns its key. Idempotent on the derived key.
List projects and their apps (surfaces). Use an app key as the `appKey` when opening a session.
Persist a discovered fact (selector / quirk / exploration / healing / auth) scoped to the app so future sessions reuse it.
Re-run a finalized case; a failure auto-files a board bug, which you can proba_diagnose (live candidate locators) then proba_patch_step to fix; a clean replay auto-resolves it.
Execute and record an API request with layered assertions (status → schema → body → sla).
Capture the open session's auth (cookies + localStorage, i.e. Playwright storageState) and store it for the app. After this, replays and new sessions for the app start ALREADY logged in — gated routes work without re-login steps. Log in once (manually or via an account), then call this. Optional `name` to keep more than one (e.g. per role).
Open/resume a QA session for an app. `appKey` is the app/surface key (see proba_project_list / proba_app_create). Returns how much is already known (selectors) so you resume instead of re-exploring.
Begin recording a new test case.
No error handling or recovery guidance. Tools lack descriptions of failure modes, retryability, or next steps. E.g., proba_session_open could fail if appKey is invalid, no guidance on how to recover. No categorization of errors (retryable vs user-fixable vs fatal).
proba_act parameter 'target' has nested properties (strategy, value, name, exact) but descriptions are minimal. 'strategy' enum includes 'css' but description says 'Locators must be role/text/testid, positional css is rejected.' Contradiction. Parameter relationships undocumented (when is 'name' vs 'value' used?).