Loading ToolBench
TestFlow Orchestrator is an MCP server that bridges test discovery and execution. It specializes in template-based test management, real-time execution monitoring, and comprehensive result tracking.
Server has 7 tools with significant definition gaps. Tool names lack consistent verb-noun structure (getAllContent, runningATemplate are awkward). Most descriptions are present but verbose and lack actionable guidance. Input schemas exist but lack parameter-level descriptions and type constraints. No output schemas documented. Error handling is absent. Security concerns: file upload tool (bulkUploadTestDefinitionFiles) lacks path sanitization details and permission checks. Overall, definitions are below production baseline.
Upload multiple test files in a single operation. Supports uploading multiple feature files and test data files, saves files to appropriate directories based on type (feature or test_data), performs path sanitization for security, automatically creates parent directories, provides detailed success/failure reporting for each file, and returns operation statistics with unique operation ID.
List all available test templates and metadata.
Retrieve complete content of all feature files and test data templates. Scans feature files from the data/manager/lib/features/ directory, retrieves test data files from the data/manager/lib/test_data/ directory, provides both relative and absolute file paths, includes full content of each file with binary detection.
Get comprehensive execution information including test execution status and timeline, XML result analysis with pass/fail determination, database records of all associated tests, step-by-step execution details, and test summary.
Retrieve all test executions with optional filtering by status (running, pending, completed, failed). Provides progress tracking with completion statistics.
Tool naming inconsistency: 'runningATemplate' is grammatically awkward and unclear. Should be 'executeTemplate' or 'runTemplate'. 'getAllContent' lacks verb clarity (get what? all what?). Inconsistent naming forces LLMs to reason about intent.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. E.g., runTestsFromTemplate returns 'wsUrl' but no schema shows structure, type, or required fields.
Parameter descriptions missing or incomplete. 'test_infos' array lacks item schema details (what fields required in each object?). 'status' enum values not documented in parameter description. LLMs cannot validate inputs.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | <=2025-11-25 | v2 |
Execute the specified test template with given parameters.
Execute tests using predefined templates or custom scripts. Supports feature templates and custom Gherkin scripts, data-driven testing with template substitution, multiple browser support (Chrome, Firefox, Safari), environment-specific execution (dev, staging, production), mock mode for testing without actual execution, and parallel test execution with unique directory management. Returns a wsUrl field with a WebSocket URL to monitor execution progress.
Security: bulkUploadTestDefinitionFiles claims 'path sanitization' but no validation logic visible in source. File upload without explicit permission checks or rate limits risks abuse. No error handling for malicious paths.
No error handling guidance. Tools lack recovery hints (e.g., 'If execution not found, call getExecutionList() to discover valid IDs'). Errors will not guide LLM to self-correct.