MCP server for TestRail integration providing tools to manage test cases, projects, runs, results, and datasets via the TestRail API
The server has 26 tools with schemas and descriptions visible in the source code. All tools are explicitly registered using FastMCP decorators. However, there are significant quality gaps: (1) Descriptions are often too brief (many under 50 chars) and lack guidance on when to use each tool vs alternatives; (2) Output schemas are not documented, we can see parameter inputs but no schema for what each tool returns, forcing LLMs to guess; (3) Parameters lack detailed constraints (e.g., status_id values 1-5 are documented inline but not as enums); (4) No error handling guidance, tools return Dict/List but don't explain failure modes or recovery paths; (5) Destructive operations (delete_project, delete_case, delete_run, delete_dataset) have no confirmation/dry-run pattern documented; (6) Parameter descriptions are inconsistent in detail, some are precise, others generic. The tools cover the TestRail domain comprehensively but execution quality is average.
Add a new test case
Add a new dataset
Add a new project
Add a new result for a test
Add multiple results for a run
Add results for specific cases in a run
Add a new test run
Output schemas are completely undocumented. Every tool's description says what it does but never states what fields/structure it returns. LLMs cannot plan downstream tool calls or extract needed data without knowing the output schema.
Four destructive operations (delete_project, delete_case, delete_run, delete_dataset) lack confirmation/dry-run patterns. Agents cannot safely invoke them without risk of accidental data loss. No recovery guidance in error responses.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 48 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 40 | - | v1 |
Close a test run
Delete a test case
Delete a dataset
Delete a project
Delete a test run
Get a test case by ID
Get all test cases for a project/suite
Get a dataset by ID
Get all datasets for a project
Get a project by ID
Get all projects
Get all results for a test
Get all results for a run
Get a test run by ID
Get all test runs for a project
Update an existing test case
Update an existing dataset
Update an existing project
Update an existing test run
Tool descriptions for simple getters (get_project, get_projects, get_case, get_cases, get_run, get_runs, get_results, get_dataset, get_datasets) are under 50 characters and lack guidance on when to use each variant or what to expect in the response.
Parameter constraints (e.g., status_id must be 1-5 for passed/blocked/untested/retest/failed) are documented in descriptions but not declared as enums. LLMs cannot auto-select valid options and must parse text descriptions, inviting hallucinated values.
No error handling guidance in any tool. Responses are documented as 'Dict' or 'List[Dict]' with no explanation of failure modes (e.g., 404 if project not found, 403 if unauthorized). LLMs cannot know what to do on failure or how to recover.
Batch operations (add_results, add_results_for_cases) accept arrays but do not document whether partial success is possible or if all-or-nothing semantics apply. No guidance on per-item error handling.
Many tools accept optional parameters (e.g., suite_id in get_cases, milestone_id in add_run) but do not document what happens when they are omitted. Does the tool error, filter, or default? Ambiguity forces LLMs to guess.
No documentation of pagination for list operations (get_projects, get_cases, get_runs, get_results_for_run, get_datasets). If these return thousands of items, the context window will be exhausted. Missing: page/offset, limit, total_count, next_cursor in responses.