Loading ToolBench
A Model Context Protocol (MCP) server that provides TestRail integration tools for AI assistants like Cursor
TestRail MCP server exhibits significant definition quality gaps across naming, descriptions, and schema completeness. While 19 tools are registered with basic schemas, parameter descriptions are sparse or missing context, and many tools lack guidance on prerequisites and error recovery. The add_case tool stands out with extensive workflow documentation but is atypical; most other tools have minimal descriptions. No tool annotations (readOnlyHint/destructiveHint) are present despite clear write operations. Output schemas are undocumented for all tools. Error handling provides generic categorization but lacks specific recovery guidance. The server registers tools across three separate files (http-server.ts, server.ts, simple-server.ts) with inconsistent detail levels, suggesting incomplete tooling strategy.
Add an attachment to a TestRail test case.
Create a new TestRail test case. REQUIRED WORKFLOW (follow exactly to avoid errors): 1. Call get_cases with section_id and limit=1 to see an existing case's field format 2. Call get_case_fields to find ALL required fields (configs[].options.is_required=true) 3. Include ALL required custom fields matching the format from step 1 REQUIRED CUSTOM FIELDS (vary by project - check is_required flag): - custom_automation_type: integer (e.g., 0=None, 1=Automated) - custom_case_complexity: integer (e.g., 0=Simple, 1=Medium, 2=Complex) - custom_version: integer (project-specific version ID) TEST STEPS - Two mutually exclusive formats exist (check existing cases to see which one the project uses): Format A - Separated steps (type_id=10, most common): "custom_steps_separated": [ { "content": "Step 1 action", "expected": "Step 1 expected result" }, { "content": "Step 2 action", "expected": "Step 2 expected result" } ] Format B - Simple text fields (type_id=3): "custom_steps": "Step 1\nStep 2\nStep 3" "custom_expected": "Expected result text" IMPORTANT: Do NOT mix formats. Check an existing case in the target section to see which format is used. HTTP 400 ERROR: Means missing required fields OR wrong field format. Check existing cases first. Section resolution: Provide section_id directly, or project_id (+ suite_id for multi-suite projects) to auto-resolve.
Add a test result to a TestRail test.
Simple echo tool for testing
Missing output schemas for all tools. LLMs cannot determine what fields to expect from responses, forcing unstructured parsing and risking missed chaining data (IDs needed for downstream tool calls).
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear write operations (add_case, update_case, add_attachment_to_case, update_run, update_test, add_result). LLMs cannot distinguish safe reads from potentially destructive writes without explicit hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 53 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 41 | - | v1 |
Fetch a TestRail test case by ID.
Get all available custom fields for test cases in a TestRail project.
Get test cases for a specific TestRail project/suite/section.
Get details for a specific TestRail project by ID.
List all TestRail projects.
Get details for a specific TestRail test run by ID.
Get test runs for a specific TestRail project.
Get test sections for a specific TestRail project/suite.
Get details for a specific TestRail test suite by ID.
Get all test suites for a specific TestRail project by ID.
Get details for a specific TestRail test by ID.
Get tests for a specific TestRail test run.
Update a TestRail test case by ID with new field values.
Update a TestRail test run by ID with new field values.
Update a TestRail test by ID with new field values.
Generic, minimal parameter descriptions across most read tools (get_projects, get_suites, get_suite, get_sections, update_test, get_case_fields). Descriptions like 'Get all TestRail projects' do not explain when to use vs similar tools or what context they provide.
add_case has extensive inline workflow documentation (411 chars of required steps embedded in description), but this pattern is not idiomatic MCP. Workflow complexity should be surfaced via get_case_fields discovery or demonstrated via examples/guides outside tool descriptions. Inline procedural text bloats descriptions and invites parsing errors.
Parameter descriptions lack constraints and format guidance. E.g., 'Section ID where the case will be created' (in add_case) does not hint that it can be omitted if project_id provided. 'Test status ID (e.g., 1=Passed, 5=Failed, 3=Untested)' in add_result shows examples as free text rather than enum constraints. Enums or explicit min/max are absent.
Error handling in http-server.ts provides categorization (auth, not_found, rate_limited, server, network) but lacks actionable guidance. E.g., 'Rate limited by TestRail; try again later' does not indicate when to retry or backoff. No per-tool error-specific messaging. Errors do not suggest next steps or available alternatives.
No pagination limits documented or enforced. get_cases and get_tests accept limit and offset parameters, but tool descriptions do not specify max results per request or cap at a reasonable default (e.g., 50). Large result sets can exhaust context windows.
Credential management: TESTRAIL_USER and API_KEY are passed as env vars but error messages reference them by name (see http-server.ts line 'check TESTRAIL_USER/API_KEY'). This exposes secret names in logs and agent context, violating secret-injection pattern. Do not mention secret names in error messages.