An MCP server for testing and analyzing OpenAPI/Swagger APIs with deterministic test plan generation, semantic search over contract documentation, and policy-driven execution with safety constraints
This is a specialized API testing and exploration tool with 13 well-named tools and mostly good descriptions. Naming follows verb_noun convention consistently (ApiGenerateTestPlan, ApiListOperations, ApiDescribeOperation, etc.). Most tools have detailed descriptions (avg ~120 chars, within the 10-1024 char baseline). However, there are critical gaps: (1) Input schemas are visible in the codebase but not all parameters are shown with explicit type metadata in the excerpts, only operationId and a few others are clearly typed as 'string'; (2) Parameters in several tools (e.g., pathParamsJson, queryParamsJson, headersJson, bodyJson in ApiCallOperation) are documented as JSON strings but lack explicit format constraints, regex patterns, or guidance on structure; (3) Output schemas are not explicitly documented, the code returns anonymous objects with implicit structure; (4) Error handling is basic, no recovery guidance, no categorization of retryable vs fatal errors; (5) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear semantic intent. The domain (API contract testing) is well-served by the tool set, but the execution lacks production-grade rigor. Schema completeness is moderate, types are present but incomplete in documentation.
Ask Azure AI a grounded question about the loaded API using only retrieved OpenAPI evidence.
Build or execute an OpenAPI operation by operationId. The current policy controls methods, target URLs, network access and dry-run/live behaviour.
Clear any configured API authentication from this process.
Describe an OpenAPI operation (method, path, params, request body, responses, security) by operationId.
Generate a deterministic, constraint-driven API test plan for an operationId using the OpenAPI contract. Includes parameters, schema bounds and concrete edge-case inputs.
Get the current in-memory execution policy.
List operations in the loaded OpenAPI document with operation ID, method, path and summary.
Output schemas not documented. Tools return anonymous C# objects with implicit structure (e.g., ApiGenerateTestPlan returns object with operationId, method, path, testCases). LLMs need explicit field definitions to plan downstream calls and extract data reliably. This forces agents to infer structure from execution rather than reading a schema.
JSON-string parameters lack format constraints. ApiCallOperation accepts pathParamsJson, queryParamsJson, headersJson, bodyJson as bare strings with description 'JSON object of...' but no schema constraint (minLength, maxLength, pattern, examples). This invites malformed JSON and gives LLMs no guidance on structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 51 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 35 | - | v1 |
Load an OpenAPI/Swagger definition from a URL or local file. Parses it, builds semantic operation/schema/security evidence and replaces the in-memory vector index atomically.
Reset execution policy to safe dry-run, deny-by-default settings.
Search the loaded OpenAPI contract semantically and return the most relevant operation/schema/security evidence without calling the chat model.
Set the base URL used for executing API requests. Overrides servers[] in the OpenAPI spec for this process only.
Set a Bearer token in memory for API requests made by this MCP process.
Update the in-memory execution policy. Disabled unless the MCP process was started with policy mutation explicitly enabled.
No tool annotations. Tools like ApiCallOperation (WRITE risk), ApiSetPolicy (REVERSIBLE), ApiSetBearerToken (REVERSIBLE) are not tagged with destructiveHint, idempotentHint, or readOnlyHint. MCP 2.0 tool annotations help agents understand safety implications without parsing descriptions.
Error handling is minimal. Tools validate inputs (e.g., 'operationId is required') with basic ArgumentException/InvalidOperationException but provide no recovery guidance. Example: 'OperationId not found: {operationId}' tells the agent nothing, should be 'OperationId not found: {operationId}. Call ApiListOperations() to see available IDs.'
Parameter descriptions are generic for complex inputs. topK in ApiSearchContract and ApiAskContract is described only as 'Maximum number of results to return (default 8, clamped 1-20)' but does not explain what the retrieval logic optimizes for (semantic similarity? recency? relevance ranking?). Agents cannot reason about appropriate values without this context.
Bearer token parameter is a security anti-pattern. ApiSetBearerToken accepts 'token' as a direct tool parameter. Per pattern:secret-injection, credentials must use server-side injection. If an agent logs this call or it appears in traces, the token leaks.