Loading ToolBench
MCP server for interacting with SmartBear Products including Bugsnag, Reflect, Swagger, PactFlow, Zephyr, and QTM4J
SmartBear MCP provides 26 tools across Pactflow, QMetry, and related services. While tool names are generally action-oriented (generate, review, getProviderStates, canIDeploy, etc.), there are significant definition quality gaps: (1) Input schemas are declared as generic object types with only 'description' fields (e.g., {'type':'object','description':'GenerationInputSchema'}) rather than properly expanded JSON Schema with explicit properties, types, and constraints. (2) Many tool descriptions lack actionable context, they state WHAT the tool does but omit WHEN to use it or what prerequisites exist. (3) Parameter descriptions in the schema payloads are not visible in the source, only schema class names are referenced (GenerationInputSchema, RefineInputSchema, etc.), making it impossible to verify parameter documentation. (4) No output schema documentation is visible. (5) Tool composition has minor issues: tools like checkAIEntitlements appear twice (tools 6 and 9), suggesting possible copy-paste errors. (6) Error handling guidance is absent from tool descriptions. On the positive side: (1) Tools follow verb_noun naming (generate, review, get*, fetch*, create, update, link, export, import). (2) Descriptions are generally between 50-300 characters, within the acceptable range. (3) Risk classification (READ_ONLY vs WRITE) is declared for all tools, showing awareness of destructive operations. (4) Pagination parameters (pageNumber, pageSize, page, limit) are present where appropriate. The average per-tool score drops to ~52 due to missing parameter details and schema expansion.
Create a new defect/issue internally in QMetry.
Execute a quality gate report by forwarding the request to the backend analytics engine and returning the results.
Export HTML content as a downloadable report file via the backend.
Fetch automation status
Fetch the quality gate configuration for a project and AI agent, including assessment scope and gate criteria.
Fetch issues linked to test case
Fetch issues list
Input schemas are opaque generic objects without expanded property definitions. Tools declare inputSchema as {'type':'object','description':'SchemaName'} rather than exposing properties, types, constraints, and per-parameter descriptions. This makes it impossible for LLMs to understand what parameters are valid or required.
Many tool descriptions are extremely generic or missing. 'Fetch metrics across the entire workspace' (getMetrics), 'Fetch automation status' (FETCH_AUTOMATION_STATUS), 'Fetch issue details' (FETCH_ISSUE_DETAILS) provide no actionable context about when to call these tools or what they return.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 55 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 47 | - | v1 |
Fetch issue details
Fetch issue executions
Fetch linked issues by test case run
Fetch QMetry projects list including projectID, name, projectKey, isArchived, viewIds and folderPath needed for other operations
Fetch QMetry project information including viewId and folderPath needed for other operations
Fetch QMetry releases and cycles from the current project
Import/Publish automation test results from TestNG, JUnit, Cucumber, Robot, HPUFT, or QAF frameworks into QMetry
Link issues to test case run
Set current QMetry project for your account
Update issue
Performs a comprehensive compatibility check to determine whether a specific version of a service (pacticipant) can be safely deployed into a given environment. It analyzes the complete contract matrix of consumer-provider relationships to confirm that all required integrations are verified and compatible.
Check PactFlow AI entitlements
Check your PactFlow AI entitlements and credit balance if you encounter 401 Unauthorized errors or permission/credit issues when using PactFlow AI features.
Generate Pact tests using PactFlow AI. You can provide one or more of the following input types: (1) request/response pairs for specific interactions, (2) code files to analyze and extract interactions from, and/or (3) OpenAPI document to generate tests for specific endpoints. When providing an OpenAPI document, a matcher is required to specify which endpoints to generate tests for.
Retrieve the comprehensive contract verification matrix that shows the relationship between consumer and provider versions, their associated pact files, and verification results stored in the Pact Broker or Pactflow. The matrix provides detailed visibility into which consumer and provider versions have been successfully verified against each other, and highlights failures with detailed information about the cause.
Fetch metrics across the entire workspace
Retrieve the states of a specific provider. A provider state in Pact defines the specific preconditions that must be met on the provider side before a consumer-provider interaction can be tested. It sets up the provider in the right context—such as ensuring a particular user or record exists—so that the provider can return the response the consumer expects. This makes contract tests reliable, repeatable, and isolated by injecting or configuring the necessary data and conditions directly into the provider before each test runs.
Fetch metrics for all teams
Review Pact tests using PactFlow AI. You can provide the following inputs: (1) Pact tests to be reviewed along with metadata
Duplicate tool name: checkAIEntitlements appears twice (tools 6 and 9). This causes ambiguity and prevents the LLM from selecting the intended tool consistently.
No output schema documentation is visible. Tools do not declare what fields will be returned, their types, or how they chain to other tools. This forces LLMs to guess at response structure and makes multi-step tool composition unreliable.
No error handling guidance in tool descriptions. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do if a call fails. Error responses lack actionable recovery steps.
FETCH_* tools (14 tools) use SCREAMING_SNAKE_CASE naming, which is inconsistent with camelCase naming used by Pactflow tools (get*, fetch*, create, update, etc.). Naming inconsistency increases LLM confusion when selecting between similar operations.
Tool composition: Parameter documentation missing from many tools. For example, LINK_ISSUES_TO_TESTCASE_RUN has parameter schema 'LinkIssuesToTestcaseRunArgsSchema' (not expanded), so the LLM cannot determine what issueIds or testcaseRunIds it expects, or whether they are required.