Loading ToolBench
MCP server for API automation testing, providing tools for graph-based workflow management, archive inspection, documentation generation, and domain knowledge browsing
AAT provides 14 well-named, read-only tools with consistent verb-noun naming (list_*, inspect_*, get_*, analyze_*, diff_*). All tools have descriptions (avg ~150 chars, within baseline 194). Input schemas are present and typed. However, parameter descriptions are sparse or missing in several tools (e.g., get_sample_response has 4 params but only 2-3 have substantive descriptions; path and shape lack detail on expected formats). Output schemas are not documented, LLMs cannot predict response structure for chaining. Error handling is minimal; no recovery guidance or actionable error messages visible in code. All tools are READ_ONLY, reducing security risk but limiting utility. Tool composition is sound (each does one thing), but missing pagination guidance for list_* tools and no documented limits on result sizes.
Analyze a failed run archive and provide failure-focused diagnostics with suggested next steps.
Side-by-side comparison of two run archives showing outcome, status, duration, and output differences.
Show full detail for a domain concept: description, constraints, examples, and related types/pools
Generate a Markdown documentation skeleton for a node, including graph metadata, input/output tables, and placeholder sections. Returns text — does not write to disk.
Show merged documentation for a graph node: graph metadata, user-written Markdown docs, and OAS summary. Works even without docs — falls back to graph metadata only.
Get a sample API response for an operation from run archives: the newest successful response, or the newest failed one when no run succeeded. Shows the response body, status code, and source run. `path` narrows the body with a gjson path, and `shape` returns the body's structure instead of its values (each path with its type, array sizes, and a sample value), which suits large responses. Useful for understanding response shapes and extract rules.
Output schemas not documented. LLMs cannot predict response structure (fields, types, nesting) for downstream tool chaining or data extraction. Visible in all 14 tools.
get_sample_response parameters 'path' and 'shape' lack substantive descriptions. 'path' mentions gjson but does not explain format or constraints; 'shape' is vague on what 'structure' means. LLMs cannot reliably use these params.
list_* tools (list_archives, list_recent_failures, list_concepts, list_types, list_value_pools) accept 'limit' but no pagination (offset, cursor, next_token). No guidance on max result size or whether results are truncated. Large result sets risk context window exhaustion.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Show merged documentation for every node in a workflow chain, traced backward from the goal node.
Show detailed Markdown view of a run archive including per-step request/response data, assertions, selections, and value resolutions.
List recent run archives from the archive directory, showing run ID, timestamp, outcome, and duration.
List all domain concepts with descriptions and applicable fields
List recent failed run archives, showing run ID, timestamp, outcome, and error summary. Skips passed runs.
List all domain type definitions with format and field info
List graph nodes that do not have a corresponding Markdown doc file. Shows which nodes need documentation.
List all value pools with type, sample values, and total count
No error handling guidance visible in tool definitions. Tools do not document what errors are possible, whether they are retryable, or what the LLM should do next (e.g., 'run_id not found, try list_archives() first').
Tool descriptions do not explain WHEN to use each tool or how they relate. E.g., difference between inspect_archive (detailed view) and analyze_failure (failure-focused) is unclear. LLMs may pick the wrong tool.