Loading ToolBench
An MCP server that exposes TestRail test management APIs through a semantic search interface. Provides access to 101+ internal tools for managing test cases, runs, results, projects, and attachments, exposed via search_tools, get_categories, get_tools_by_category, and execute_tool gateway methods.
The TestRail MCP server exposes 13 tools with mixed quality. Strengths: most tools have detailed, LLM-optimized descriptions (100-300+ chars) with clear use cases, examples, and 'might lead to' chaining hints. All tools have named parameters with types and descriptions. Weaknesses: (1) Tools 1-3 are discovery/gateway tools (search_tools, get_categories, get_tools_by_category) that wrap 100+ internal tools not visible in the source, we cannot verify schemas for those internal tools, so overall composability is opaque. (2) Tool 4 (execute_tool) has a generic 'parameters' field that is an untyped object, LLMs cannot infer what keys/values are valid without calling search_tools first, creating a circular dependency. (3) No tool declares enums for constrained inputs (e.g., field type IDs 1-11 for add_case_field). (4) Output schemas are not documented for any tool, we see descriptions of what they return but no formal schema. (5) No tools declare error handling or recovery guidance. (6) No tool has idempotent, destructive, or readOnly hints. (7) Security: no validation shown for parameter injection (e.g., caseId is an integer, but no range given; sectionId likewise). (8) Pagination parameters on tools 5-9 are optional but not constrained (no min/max for limit/offset). Composition is strong within the attachment/BDD/field tools, but fragmented across the gateway pattern.
Imports/uploads a BDD scenario to a section from .feature file content in Gherkin syntax. Creates a new test case from the BDD scenario with Feature, Scenario, Given/When/Then steps. The feature file content should follow Gherkin syntax with Feature, Background, Scenario, Examples, and Rules. Returns the created test case with BDD content stored in custom fields. **When to use:** Use this tool when you need to import BDD scenarios from external tools into TestRail, sync Cucumber/SpecFlow .feature files with TestRail, automate test case creation from BDD specifications, or migrate BDD scenarios from other systems. **Might lead to:** get_bdd (to verify import), get_case (to view created test case), update_case (to modify the imported scenario). **Example prompts:** - "Import BDD scenario to section 188" - "Upload this .feature file content to section 50" - "Create test case from BDD scenario in section 'Login Tests'"
Creates a new custom field for test cases. Allows extending test case data structure with organization-specific fields. Supports 11 field types: String, Integer, Text, URL, Checkbox, Dropdown, User, Date, Milestone, Steps, Multiselect. Can be configured as global (all projects) or project-specific. Field names are automatically prefixed with "custom_" (e.g., "my_field" becomes "custom_my_field"). **When to use:** Use this tool when you need to capture additional test case information not covered by built-in fields, standardize custom data collection across projects, add domain-specific metadata to test cases, or support specialized testing workflows (e.g., regulatory compliance, performance metrics). **Might lead to:** get_case_fields (to verify creation), add_case or update_case (to populate new field). **Example prompts:** - "Create a dropdown custom field called 'Affected Component' with options: Frontend, Backend, Database" - "Add a text custom field 'Test Data Requirements' for all projects" - "Create a multiselect field 'Tags' with options: Smoke, Regression, Integration" - "Add a user field 'Test Designer' to track who designed each test"
execute_tool accepts a generic 'parameters' object with no schema constraints. The tool description says 'flat key-value map matching the parameter names' but does not declare which keys are valid, their types, required fields, or constraints. LLMs cannot validate input before calling execute_tool and may pass malformed parameters.
100+ internal tools exist but are not visible in source code. The server uses a gateway pattern (search_tools + execute_tool) to expose them. Actual schemas, descriptions, and error handling for those tools cannot be verified. This makes the server's true surface area opaque and prevents assessment of the internal tool quality. Composition across the boundary is unclear.
No output schemas documented for any tool. Descriptions say 'Returns attachment metadata including ID, filename, size...' but do not provide a formal schema showing field types, nested structure, or required fields. LLMs cannot plan downstream calls or extract fields reliably.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Executes a specific TestRail tool by name with the provided parameters. Use search_tools, get_categories + get_tools_by_category to discover tool names and their required parameters before calling this tool. Parameters should be provided as a flat key-value map matching the parameter names returned by the discovery tools. For example: - execute_tool(toolName: "get_case", parameters: {"caseId": 123}) - execute_tool(toolName: "add_result", parameters: {"testId": 5, "statusId": 1, "comment": "Passed"}) - execute_tool(toolName: "get_cases", parameters: {"projectId": 1, "suiteId": 5})
Retrieves all attachments associated with a test case. Returns attachment metadata including ID, filename, size, upload date, and user. Supports pagination via limit and offset parameters (TestRail 6.7+). Requires TestRail 5.7 or later. **When to use:** Use this tool when you need to view all files attached to a test case, download case documentation or reference materials, audit attachment history, verify uploaded files, or prepare for attachment migration/cleanup. **Might lead to:** get_attachment (to retrieve specific attachment details), delete_attachment (to remove files). **Example prompts:** - "Show me all attachments for test case 123" - "List files attached to case 456" - "What attachments are on test case 789?"
Retrieves all attachments associated with a test plan. Returns attachment metadata including ID, filename, size, upload date, and user. Supports pagination via limit and offset parameters (TestRail 6.7+). Requires TestRail 6.3 or later. **When to use:** Use this tool when you need to view all files attached to a test plan, access plan-level documentation, audit plan attachments, verify uploaded files, or prepare for attachment migration/cleanup. **Might lead to:** get_attachment (to retrieve specific attachment details), delete_attachment (to remove files). **Example prompts:** - "Show me all attachments for test plan 10" - "List files attached to plan 25" - "What documents are attached to test plan 50?"
Retrieves all attachments associated with a specific test plan entry. A plan entry represents a test run configuration within a test plan. Returns attachment metadata including ID, filename, size, upload date, and user. Requires TestRail 6.3 or later. **When to use:** Use this tool when you need to view files attached to a specific plan entry/configuration, access entry-specific documentation, audit entry attachments, or verify uploaded files. **Might lead to:** get_attachment (to retrieve specific attachment details), delete_attachment (to remove files). **Example prompts:** - "Show me attachments for plan 10 entry 'abc123'" - "List files attached to plan entry 5 in plan 20" - "What attachments are on the Chrome configuration entry?"
Retrieves all attachments associated with a test run. Returns attachment metadata including ID, filename, size, upload date, and user. Supports pagination via limit and offset parameters (TestRail 6.7+). Requires TestRail 6.3 or later. **When to use:** Use this tool when you need to view all files attached to a test run, access run-level documentation or logs, audit run attachments, verify uploaded files, or prepare for attachment migration/cleanup. **Might lead to:** get_attachment (to retrieve specific attachment details), delete_attachment (to remove files). **Example prompts:** - "Show me all attachments for test run 100" - "List files attached to run 250" - "What logs are attached to test run 500?"
Retrieves all attachments associated with a specific test (test instance in a run). Returns attachment metadata including ID, filename, size, upload date, and user. Requires TestRail 5.7 or later. **When to use:** Use this tool when you need to view files attached to a specific test execution, access test-specific screenshots or logs, audit test attachments, or verify uploaded files. **Might lead to:** get_attachment (to retrieve specific attachment details), delete_attachment (to remove files). **Example prompts:** - "Show me all attachments for test 1000" - "List files attached to test 2500" - "What screenshots are on test 5000?"
Exports a BDD scenario from a test case as a .feature file in Gherkin syntax. Returns the complete feature file content including Feature, Background, Scenario, Given/When/Then steps, Examples, and Rules. Useful for syncing TestRail test cases with BDD testing frameworks (Cucumber, SpecFlow, Behave). **When to use:** Use this tool when you need to export test cases for BDD test automation, sync TestRail scenarios with Cucumber/SpecFlow projects, generate .feature files for CI/CD pipelines, or migrate BDD scenarios to external testing frameworks. **Might lead to:** add_bdd (to re-import modified scenarios), update_case (to modify the test case). **Example prompts:** - "Export BDD scenario from test case 2133" - "Get the .feature file for case 100" - "Download BDD scenario for test case 'Login validation'"
Retrieves all available test case custom field definitions. Returns field metadata including type, label, description, configurations, and project applicability. Custom fields can be global (all projects) or project-specific. Field types include: String, Integer, Text, URL, Checkbox, Dropdown, User, Date, Milestone, Steps, Multi-select. **When to use:** Use this tool when you need to understand what custom fields are available, check field configurations before creating/updating test cases, discover field system names for API usage, audit custom field setup across projects, or verify field types and options. **Might lead to:** add_case_field (to create new field), add_case or update_case (to use fields). **Example prompts:** - "Show me all custom case fields" - "What custom fields are available for test cases?" - "List all case field definitions and their types" - "What's the system name for the 'Affected Component' field?"
Returns all available tool categories with the number of tools in each. Use this to explore what the TestRail MCP server can do, then call get_tools_by_category() to see the tools in a specific category. Example response: {"categories": [{"name": "test-cases", "toolCount": 6}, {"name": "test-runs", "toolCount": 6}, ...]} Alternatively, use search_tools() for natural language discovery.
Returns all tools in a specific category with full details including: - name: the tool identifier to use with execute_tool - description: what the tool does - keywords: related search terms - examples: usage examples - parameters: full parameter specifications (name, type, description, required, defaultValue) Use get_categories() first to see available categories, then call this tool with a category name. After finding the right tool, use execute_tool to run it. Alternatively, use search_tools() for natural language discovery.
Searches for available TestRail tools matching a natural language query. Returns a ranked list (max 20) of matching tools with full details including: - name: the tool identifier to use with execute_tool - description: what the tool does and when to use it - category: the tool's functional category - keywords: related search terms - examples: usage examples showing how to call via execute_tool - parameters: full parameter specifications (name, type, description, required, defaultValue) Use this for fuzzy or natural language queries like 'clone a test case' or 'add test result'. Alternatively, use get_categories() and get_tools_by_category() for structured browsing. After discovering the right tool, use execute_tool to run it.
Pagination parameters (limit, offset) on tools 5-9 (attachment retrieval) lack constraints. No minimum/maximum specified for limit (e.g., 1-100) or offset (e.g., >= 0). LLMs may pass absurd values like limit=999999 or negative offset.
add_case_field accepts a generic 'caseField' object parameter with no schema. The description mentions field type IDs (1-11) and keys like 'type_id', 'name', 'label' but does not provide a formal schema or enum for type_id. LLMs cannot validate or auto-complete field configurations.
No error handling or recovery guidance in any tool description. When a tool fails (e.g., invalid caseId, permission denied, API timeout), the description does not explain what the LLM should do next or how to distinguish retryable vs fatal errors.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present. Tools like add_bdd and add_case_field are clearly destructive but carry no hint. Tools like get_attachments_for_case are read-only but unmarked. This prevents agents from making informed planning decisions.
Integer parameters (caseId, planId, runId, testId, limit, offset) lack range constraints. Descriptions do not specify valid ranges, minimum values, or maximum values. LLMs may pass invalid IDs like caseId=-1 or caseId=999999999.