Loading ToolBench
MCP server wrapper for Specmatic contract testing, providing tools for running contract tests, resiliency tests, managing mock servers, and checking backward compatibility in OpenAPI specifications
The Specmatic MCP server defines 4 tools with reasonable structure but significant gaps in description quality and schema completeness. All tools have input schemas present (meeting minimum requirement), but descriptions are inconsistent and error handling lacks recovery guidance. Tool naming follows verb_noun convention (run_*, manage_*, backward_compatibility_check) which is correct. However, parameter descriptions are sparse and output schemas are not documented. The manage_mock_server tool has conditional parameter requirements that are inadequately documented, forcing LLMs to reason about dependencies. No evidence of error classification or recovery guidance in responses.
Check for breaking changes in OpenAPI specifications using Specmatic's git-based analysis. Works with relative paths and automatically detects git repository context. Example: targetPath='products_api.yaml'
Manage Specmatic mock servers - start, stop, or list running servers for frontend development. Supports complete mock server lifecycle management.
Run Specmatic contract tests against an API using OpenAPI specification
Run Specmatic resiliency tests with boundary condition testing against an API using OpenAPI specification. This enables SPECMATIC_GENERATIVE_TESTS to test how the API handles contract-invalid requests
Output schemas not documented. LLMs cannot predict what fields will be returned, complicating downstream tool chaining and data extraction. run_contract_test, run_resiliency_test, manage_mock_server, and backward_compatibility_check all lack return type documentation.
manage_mock_server has conditional required parameters (openApiSpec required only for 'start', port required for 'start' and 'stop') but this dependency is not explicitly documented in parameter descriptions. LLMs will struggle to determine which parameters to include for each command variant.
No error handling guidance in tool descriptions. run_contract_test and run_resiliency_test do not document what errors are retryable, what user action is needed, or recovery paths. backward_compatibility_check lacks guidance on repository setup requirements.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | D | 56 | 2026-07-28+ | v2 |
| 2026-03-09 | C | 61 | 2024-11-05+ | v1 |
Parameter descriptions are generic or minimal. 'openApiSpec' is described as 'The OpenAPI specification content (YAML or JSON)' but lacks guidance on file size limits, required fields, or validation rules. 'apiBaseUrl' lacks validation constraints (protocol requirements, port restrictions). These sparse descriptions force LLMs to infer constraints.
backward_compatibility_check has all optional parameters but the description states 'If not provided, Specmatic will analyze all tracked specification files' without documenting required context (git repository must exist, files must be tracked). This creates silent failures.
No tool descriptions explicitly state whether operations are idempotent, have side effects, or require retries. manage_mock_server's 'start' command may fail if port is in use (documented in source code) but this constraint is not in the tool description visible to LLMs. Tool descriptions do not mention persistence or session state.