Loading ToolBench
mcp-xray demonstrates solid definition quality with 9 well-named tools following verb_noun patterns (xray_check_connection, xray_get_test_execution, xray_search_test_executions, etc.). All tools have descriptions and input schemas defined in Zod. However, several important gaps reduce the score: (1) Output schemas are not documented, the source shows tool handlers return asJsonResult() with minimal guidance on response structure, forcing LLMs to infer output fields; (2) No error recovery guidance, error handling via asErrorResult() returns basic HTTP status and details but does not guide the agent on what to do next (retry? ask user? alternative tools?); (3) Parameter descriptions could be more actionable, while present, they lack constraint details (e.g., 'maxResults' shows 1-100 range in description but not in schema; 'format' enum is clear but example values are missing for JQL syntax); (4) Tool composition could be improved, xray_create_test_execution and xray_create_test_plan are nearly identical, suggesting potential over-duplication. Strengths: Consistent naming with 'xray_' prefix, clear separation of concerns (search vs. get), good use of Zod schemas for input validation, and proper handling of optional parameters (projectKey, testExecKey, etc.). The average tool scores to 72.
Validates authentication against the configured Xray deployment (cloud or datacenter) and Jira instance. Call this first to verify credentials are working before using other tools.
Creates a new Test Execution issue in Jira/Xray. Requires a project key and summary. Optionally provide a description and additional Jira fields (e.g. labels, components).
Creates a new Test Plan issue in Jira/Xray. Requires a project key and summary. Optionally provide a description and additional Jira fields (e.g. labels, components).
Fetches a single Test Execution issue from Jira by its issue key (e.g. 'OQA10-42', 'GAP-100'). Returns the full Jira issue details. Use the 'fields' parameter to limit which fields are returned.
Fetches a single Test Plan issue from Jira by its issue key (e.g. 'GAP-12471'). Returns the full Jira issue details. Use the 'fields' parameter to limit which fields are returned.
Output schemas not documented. Tool handlers return JSON via asJsonResult() but the structure, field names, and types of responses are not specified. LLMs cannot plan downstream operations or extract chained IDs without documented response schemas.
Error responses lack recovery guidance. asErrorResult() returns isError=true with error message and status, but does not tell the LLM what to do next (retry? ask user? call an alternative tool?). A 404 should hint at search tools; a 401 should guide to credential check.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 74 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Executes a GraphQL query against Xray Cloud. Only available for cloud deployments. Use this for advanced Xray-specific queries like fetching test runs, test steps, preconditions, or test sets.
Imports automated test execution results into Xray. Supports formats: junit, nunit, xunit, robot, testng, cucumber, behave, json. Provide the full report content as a string. Optionally target a specific project or existing Test Execution issue.
Searches for Test Execution issues in Jira using filters. The issuetype filter is applied automatically. Use 'projectKey' to filter by project (e.g. 'OQA10'), 'jql' for additional JQL conditions (e.g. 'status = "Done"'), and 'orderBy' for sorting (e.g. 'created DESC'). Example: to find recent test executions in project OQA10, use projectKey='OQA10' and orderBy='created DESC'.
Searches for Test Plan issues in Jira using filters. The issuetype filter is applied automatically. Use 'projectKey' to filter by project (e.g. 'GAP'), 'jql' for additional JQL conditions, and 'orderBy' for sorting (e.g. 'created DESC'). Example: to find test plans in project GAP, use projectKey='GAP'.
Pagination limits not enforced in response. xray_search_test_executions and xray_search_test_plans accept maxResults but response cardinality is not capped, a cloud API returning 100 results will bloat the LLM context. Add explicit cap and next_cursor guidance.
Parameter constraint descriptions missing actionable details. 'maxResults' description states '1-100' in text but schema uses z.number().max(100), format and min/max bounds are scattered. Consolidate all constraints (min, max, format, regex, enum) into descriptions so LLMs understand them even if they cannot parse JSON Schema.
No distinction between system IDs and human-readable names. issueKey expects 'OQA10-42', and projectKey expects 'OQA10', but there is no guidance on whether LLMs can pass partial names ('OQA') or friendly descriptions. Document whether lookups by name are supported or if opaque IDs are required.
Tool duplication: xray_create_test_execution and xray_create_test_plan have identical schemas and nearly identical descriptions. Consider consolidating into xray_create_tracked_issue with an 'issueType' parameter to reduce tool count and cognitive load for LLMs choosing between similar operations.
JQL filter guidance incomplete. xray_search_test_executions and xray_search_test_plans accept a 'jql' parameter with a note 'Do NOT include issuetype or ORDER BY' but provide no examples of valid JQL syntax or link to Jira JQL docs. LLMs will struggle to compose valid filters without examples.