Loading ToolBench
AI-driven end-to-end testing platform with MCP-First architecture. Provides tools for managing test projects, cases, templates, and execution via MCP protocol.
OpenTester demonstrates a solid foundation with 20 tools covering project, template, test case, and execution management. All tools have descriptions and input schemas are present in most cases. However, there are consistent gaps in parameter descriptions, output schema documentation, and error handling guidance. The tool set is well-organized and uses clear verb-noun naming conventions. Descriptions are adequate (averaging ~80-120 chars) but could be more prescriptive about when to use each tool. The main weakness is incomplete parameter documentation, many parameters lack detailed descriptions explaining constraints, formats, or dependencies. No tool annotations (readonly, destructive, idempotent hints) are present despite clear risk classifications in the metadata.
Create a new test project
Create a new DSL template
Create a new test case in a project
Delete a project
Delete a template
Delete a test case
Get detailed execution results including all steps and diagnostics
Get the status of a test execution
Missing output schema documentation. No tool documents what fields are returned, data types, or structure. LLMs cannot plan downstream tool calls or extract needed data without knowing return signatures.
No tool annotations despite clear risk classifications. Destructive tools (delete_project, delete_template, delete_test_case) lack destructiveHint. None of the readonly tools (list_*, get_*) have readOnlyHint annotations. This metadata is critical for agent safety and planning.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 68 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 57 | - | v1 |
Get details of a specific project
Get details of a specific template
Get details of a test case
Instantiate a template into a test case by substituting variables
List all test projects
List all DSL templates
Start a test execution
Stop a running execution
Update project details
Update an existing template
Update a test case
Validate DSL content syntax and schema
Parameter descriptions lack constraint details. Many parameters (e.g., 'project_id', 'template_id', 'case_id') are described only as 'The X ID' without format, length, or pattern guidance. Constraint format descriptions missing for 'execution_mode' enum and DSL content validation rules.
No error guidance or recovery hints. Error responses likely lack actionable next steps. Destructive operations (delete_*) should support confirmation or dry-run patterns to prevent accidental data loss.
list_templates, list_projects lack pagination controls. No 'limit', 'offset', 'page', or 'cursor' parameters visible. Unbounded lists risk context window exhaustion and poor agent reasoning with large result sets.
create_template 'variables' parameter has nested structure but lacks type validation hints. Schema shows array of objects with name, description, required, default_value, but no guidance on which fields are required in the nested structure.
instantiate_template 'variables' parameter is described only as 'Variable values to substitute' without format guidance. Should explain structure: are these key-value pairs? Nested objects? Single flat object with var_name keys?
Composition: create_test_case and instantiate_template both produce test cases but via different paths (manual DSL vs template substitution). Tool descriptions should clarify when to use each, and whether both produce compatible outputs.