Serves a queryable, semantic map of a .NET test-automation solution over stdio (JSON-RPC / Model Context Protocol). Read-only access to impact analysis, step binding resolution, endpoint reach, lexical search, and summary statistics against an indexed code map.
TestAtlas MCP has 12 tools with comprehensive descriptions (mostly 150-300 chars, well above the 34-char p10 baseline). All tools have input schemas with typed properties. However, parameter descriptions are inconsistent, some parameters lack descriptions entirely (e.g., 'limit' in list_endpoints, list_tags, step_catalog), and several parameter descriptions are terse. Tool naming is verb-first and clear (stats, impact, search_steps, list_endpoints, resolve_step, get_scenario, get_step_definition, list_tags, step_catalog, project_dependencies), following the verb_noun convention well. Output schemas are NOT documented in the visible code, the tool handler functions return JSON strings but there is no published schema for what fields agents should expect. Error handling is minimal: the code returns JSON-RPC errors for invalid tool calls and parse failures, but tools themselves provide no domain-specific error guidance (e.g., no 'not found' recovery hints). A critical issue: resolve_step is duplicated (tools 6 and 9 are identical), wasting an agent's tool discovery budget. Security is sound (read-only tools, no secrets exposed as parameters). Composition is good, tools chain naturally (search_scenarios → get_scenario → resolve_step). Overall, this is solid exploratory/discovery tooling for a test codebase, but lacks the polish and error guidance expected of production tools.
Full detail of scenario(s) whose name contains the given text: feature, tags, kind, example-row count, file:line, and the ordered steps (keyword + text + doc-string/data-table flags). Use to read an existing scenario before writing a similar one.
Full detail of step definition(s) whose expression contains the given text: keyword, expression kind, method parameters, C# class/method/signature, file:line, and the scenarios that currently use it (usage count). Use to inspect a step before reusing or changing it.
Blast radius of a change: the test scenarios affected by changing a class, method, step definition, or endpoint. Returns the affected scenarios (feature + the connecting step text), plus step-definition and feature counts.
The HTTP endpoints/operations the suite calls, each with verb, route (real path when known), and its scenario blast radius. Highest-reach first.
List all scenario tags in the map with usage counts per tag, ordered by frequency (highest reach first). Use to understand which hooks and filters the test suite uses and how widely they apply.
resolve_step tool is defined twice (tools 6 and 9 are identical), wasting the agent's tool discovery budget and creating confusion.
Output schemas are not documented. Agents cannot see what fields to expect from tool responses (e.g., does impact return 'scenario_count' or 'scenarios'? is it 'affected_scenarios' or 'blast_radius'?). This forces agents to guess or fail mid-chain.
Many parameters lack descriptions: 'limit' in list_endpoints, list_tags, step_catalog has no description text explaining the unit (number of results), default, or constraints. Per the rubric, 100% of A+ tool params have descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
The dependency graph between projects in the solution: which projects reference which at build time, plus the semantic dependencies surfaced by class references (inheritance, composition, step-definition bindings). Use to understand the logical structure and impact of changes across the codebase.
Resolve a Gherkin step phrase to the EXISTING step definition(s) that would bind it — the same way the runner does (regex/cucumber expression, keyword-agnostic). Use this BEFORE writing a new step so an agent reuses what already exists instead of authoring a duplicate. status is 'exact' (one binding — reuse it), 'ambiguous' (several match — a conflict to resolve), or 'none' (nothing binds — returns existing step definitions ranked by shared terms, to adapt rather than duplicate). Each match returns the expression, the C# class/method, the method parameters, the argument values captured from the phrase, and file:line.
Resolve a Gherkin step phrase to the EXISTING step definition(s) that would bind it — the same way the runner does (regex/cucumber expression, keyword-agnostic). Use this BEFORE writing a new step so an agent reuses what already exists instead of authoring a duplicate. status is 'exact' (one binding — reuse it), 'ambiguous' (several match — a conflict to resolve), or 'none' (nothing binds — returns existing step definitions ranked by shared terms, to adapt rather than duplicate). Each match returns the expression, the C# class/method, the method parameters, the argument values captured from the phrase, and file:line.
Full-text search over scenarios (feature name + scenario name + step text + tags). Returns matching scenarios.
Full-text search over step definitions (expression text + method + class name). Returns matching step definitions.
Summary of the test map: project/class/method counts, class-kind breakdown, endpoints, and edge tallies.
Full catalog of all step definitions in the map, ordered by expression; includes the C# class/method/signature, file:line, usage count per definition. Use to review the vocabulary the automation already owns.
No error guidance. If a user searches for a non-existent step or scenario, tools return empty results with no recovery hint (e.g., 'Try a shorter query term' or 'Did you mean: search_steps?'). Per the rubric, errors must tell the LLM what to do next.
Pagination is minimal. list_endpoints, list_tags, step_catalog, get_scenario, get_step_definition all accept a 'limit' parameter but do not return pagination tokens (next_cursor, total_count, or has_more). Without pagination metadata, agents cannot safely iterate large result sets.
No tool annotations. Tools do not declare readOnlyHint, destructiveHint, or idempotentHint. Per the current MCP spec (2026-07-28), these annotations help agents reason about safety and replay behavior.