Loading ToolBench
The server defines 22 tools with consistent naming patterns (verb_noun convention, mostly get_/create_/update_/delete_ prefixes). All tools have descriptions and input schemas are visible in src/tools.ts. However, descriptions vary significantly in quality and depth. Many lack explicit WHEN-to-use guidance, prerequisites, and consequence statements. Output schemas are not documented in the source, responses are implicitly JSON but structure is not formally declared. Error handling exists (safeToolCall wrapper) but recovery guidance is minimal. Overall the server meets baseline standards but falls short of A-grade clarity and LLM-optimization.
create a batch discovery for a given test target. Useful to generate multiple test cases from a prompt and optional images in one go
the createEnvironment tool can create an environment for a given test target. an environment represents a specific setup or deployments for a test target. It include a test account when necsesary to login, a header configuration, a discovery url and a set of variables.
create test cases from a test plan for a given test target. Provide freeform text, related images and tag names to assign
the createTestTarget tool can create a new test target.
the deleteEnvironment tool can delete an environment for a given test target.
the deleteTestTarget tool can delete a test target.
Output schemas are not documented. While input schemas are complete with proper zod validation, the response structure for each tool is not formally declared. LLMs cannot plan downstream calls without knowing what fields to expect.
Destructive operations (deleteEnvironment, deleteTestTarget, unregisterLocation) lack confirmation or dry-run patterns. Descriptions say 'can delete' but do not warn of irreversibility or suggest a verification step. No recovery guidance.
Parameter descriptions lack constraint specifications. E.g., 'name' parameters in createEnvironment/registerLocation do not state length limits, character restrictions, or format rules. Descriptions like 'Name of the environment' are too generic.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 69 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 36 | - | v1 |
the discovery tool can create a test case on a given test target with a test case description or prompt. One can either start from the predefined url for that test case or provide a new entry point url.
the executeTests tool can trigger a set of tests for a given test target. The test target id is unique to the test target. The tests are executed on the provided url. The context object is used to provide information about the source of the test execution.
the getEnvironments tool can retrieve environments for a given test target. an environment represents a specific setup or deployments for a test target. It include a test account when necsesary to login, a header configuration, a discovery url and a set of variables.
the getTestCase tool can retrieve a test case for a given test target and test case id. A test case id is unique to the test target. The test case includes a set of interactions and assertions. it is the result of a discovery or a manual creation.
the getTestCases tool can retrieve test cases for a given test target.
the getTestReport tool can retrieve a test report for a given test target.
the getTestReports tool can retrieve test reports for a given test target.
the getTestTargets tool can retrieve all test targets.
the listPrivateLocations tool can list all private locations.
the patchTestCase tool can update a test case for a given test target.
the registerLocation tool can register a new private location.
the search tool can be used to search the octomind documentation for a given query. The search results are returned as a list of links to the documentation.
the unregisterLocation tool can unregister a private location.
the updateEnvironment tool can update an environment for a given test target.
the updateTestCaseElement tool can update an element of a test case.
the updateTestTarget tool can update a test target.
Error responses from safeToolCall wrapper return minimal guidance. A failed API call returns only error text, not actionable recovery steps. E.g., 'Error: 404' tells the LLM nothing about what to do next.
Descriptions for tools like updateEnvironment, updateTestCaseElement, and patchTestCase are vague and do not explain what can/cannot be changed or what parameters are required vs optional.
Complex tools like discovery and createBatchGeneration have many optional parameters (entryPointUrlPath, prerequisiteName, externalId, tagNames, prompt, folderName, type) but descriptions do not explain parameter relationships, dependencies, or which combinations are valid.
No pagination or result-limiting guidance. Tools like getTestCases and getTestReports do not specify whether results are paginated, capped, or unlimited. If unbounded, returning hundreds of test cases could exhaust the context window.
Tool chaining is incomplete. For example, discovery returns a test case ID, but there is no explicit statement that the ID can be passed to getTestCase, patchTestCase, or updateTestCaseElement. Implicit chaining invites agent errors.