Loading ToolBench
MCP server wrapping every `fab` command. Exposes 19 tools, all prefixed `stf_*`, that subprocess into the bundled `fab` CLI with `--json`. Handles autonomous synthetic test loops: simulation → browser flows → scoring → feedback.
18 tools with consistent naming (stf_* prefix, verb-noun pattern) and reasonable descriptions (avg ~120 chars). All tools have input schemas with type definitions and parameter descriptions. However, output schemas are undocumented, LLMs cannot predict return structure. Error handling is minimal; tools return data.ok=false for non-fatal failures but lack recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels. Parameter descriptions are adequate but lack format constraints and validation rules. Composition is sound, tools chain well via root/flow parameters, but descriptions could better explain when to call each tool vs alternatives.
Generate a single adapter stub. type ∈ {app, simulation, scoring, feedback, memory, browser, reporter, planner}.
Type-check a TS adapter file against its target interface. Reports missing methods. Returns data.ok=false on validation failure (NOT isError).
Extract discovered screen paths from behavior events.
List all visual regression baselines.
Delete all baselines — next run will re-capture from scratch.
Accept the current screenshot as the new baseline for a flow.
CI score gate — exit 1 if fabric-score.json overall is below threshold. Returns data.ok=false on threshold failure (NOT isError).
Output schemas completely undocumented. LLMs cannot predict return structure (fields, types, pagination). Violates pattern:tool and pattern:response-shaper. Each tool description should document what fields are returned.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels. stf_baseline_reset is DESTRUCTIVE, stf_seed/stf_smoke/stf_fresh/stf_orchestrate are WRITE operations. Agents cannot distinguish safe from risky calls.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | <=2025-11-25 | v2 |
Pre-flight environment + peer-dep health check. Run before onboarding or when CI is failing for opaque reasons.
Generate feedback JSON from an existing run root. Requires fabric-score.json present (run stf_score first).
Full loop iteration: seed → verify → all flows → score → feedback. Canonical entry point for orchestration.
Scaffold a new fabric.config.ts + 8 adapter stubs into a target directory. Use when onboarding a new product to STF.
Structured summary of a loop or iteration root. Returns phase, score, flows, recent behavior events, latest screenshot.
Multi-iteration loop: fresh × N with feedback-driven persona adjustments between iterations.
Compute fabric score from an existing run root. Reads fabric-score.json via the scoring adapter.
Seed simulation fixtures into a run root. Primitive — usually called via fab smoke / fresh / orchestrate.
Fastest handoff check: seed → verify → one bounded smoke flow. Use for "is everything wired correctly".
"Where am I" — show the most recent fab command outcome from ~/.fab/state.json. First call when orienting.
Fail-closed fixture verification on an existing run root.
Error handling lacks recovery guidance. Tools return data.ok=false for non-fatal failures (e.g., stf_check threshold failure, stf_adapter_validate type mismatch) but descriptions do not explain what the LLM should do next. Violates pattern:recovery-guide.
Parameter descriptions lack format constraints and validation rules. E.g., stf_seed seekers/employers/employees are numbers but no min/max specified. stf_check threshold is 0-10 but not enforced in schema. Violates pattern:constrained-input.
Descriptions do not explain when to call each tool vs alternatives. E.g., stf_smoke vs stf_fresh vs stf_orchestrate differ in scope but descriptions lack guidance on which to choose. Violates pattern:tool-description.