Server has 11 tools with explicit schemas and descriptions. Naming follows verb_noun convention (get_, search_, create_). Descriptions are present but brief (avg ~80 chars, below 194-char baseline). All tools have input schemas with type definitions and enums where appropriate. However, output schemas are not documented, LLMs cannot predict response structure. Parameters lack depth: no format constraints, no min/max bounds, no dependency documentation. Error handling is absent from tool definitions. The create_test tool is the only write operation but lacks confirmation/dry-run pattern. No tool annotations (readOnlyHint/destructiveHint) despite clear read/write distinction.
Create a new test in the specified suite
Get a specific test plan with attached items
Get all test plans for the project
Get all root-level suites for the project
Get a specific test run with detailed information
Get all test runs for the project
Get a specific suite with its child suites and tests
No output schemas documented. LLMs cannot predict response structure, forcing them to guess field names and types for downstream tool calls. This breaks tool chaining.
No tool annotations (readOnlyHint, destructiveHint). 10 read-only tools and 1 write tool lack explicit hints. Agents cannot distinguish safe from risky operations without parsing descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Get test runs for a specific test with optional date filtering
Get all tests for the project with optional filtering
Search suites and their tests by keywords, tags, labels, and other filters
Search tests by keywords, tags, labels, TQL queries, and other filters
create_test lacks confirmation/dry-run pattern. A write operation with no safeguard against accidental execution. Agents may create unwanted tests on retry or misunderstanding.
Parameter descriptions lack actionable constraints. 'Plan ID', 'Suite ID', 'Test ID' are vague. No format hints (UUID? alphanumeric?), no length bounds, no examples of valid values. LLMs cannot validate inputs.
No error handling guidance in tool definitions. If a test ID is invalid or a suite does not exist, LLMs have no recovery path. Descriptions do not hint at prerequisite lookups or fallback strategies.