Loading ToolBench
MCP server for interacting with AWS CodePipeline services
This AWS CodePipeline MCP server has moderate definition quality with mixed results across tools. Strengths: all 12 tools have descriptions and basic schemas with typed parameters. Weaknesses: descriptions are terse (avg ~25-40 chars, well below the 50-200 char production baseline of 194 chars); output schemas are completely undocumented; no input validation guidance or error recovery help; tool names lack consistent verb-noun patterns and some are generic (e.g., 'get_pipeline_details' vs the more precise 'describe_pipeline_definition'); sensitive operations like 'approve_action', 'stop_pipeline_execution' lack confirmation patterns or destructive hints; critical chaining IDs (pipelineExecutionId, stageName) are returned by some tools but response schemas are not documented, making it unclear whether downstream tools receive necessary references. Error handling is rudimentary, no guidance for retryability, user-fixable vs. fatal errors, or recovery suggestions. The server registers tools but does not enforce input constraints (e.g., no validation that authentication enum values match AWS CodePipeline's actual supported types). Pagination support is missing despite 'list_*' tools that could return large result sets.
Approve or reject a manual approval action
Create a webhook for a pipeline to enable automatic triggering
Get the full definition of a specific pipeline
Get logs for a pipeline execution
Get performance metrics for a pipeline
Get the state of a specific pipeline
Output schemas completely undocumented across all 12 tools. LLM cannot predict response structure, forcing trial-and-error reasoning. Violates pattern:tool (tool must document return type). Example: list_pipelines returns what fields? Pipeline names, ARNs, creation dates, status, tags?
Descriptions are extremely terse (avg 28-62 chars, well below 194-char baseline). Most lack context on WHEN to call, WHAT it returns, or prerequisites. Example: 'Retry a failed stage' (18 chars) does not explain that stages in running executions cannot be retried, or what happens on timeout. Violates pattern:tool-description.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | F | 45 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 50 | - | v1 |
List executions for a specific pipeline
List all CodePipeline pipelines
Retry a failed stage
Stop a pipeline execution
Tag a pipeline resource
Trigger a pipeline execution
No input validation guidance. Parameters lack constraints (e.g., 'reason' in stop_pipeline_execution has no character limit; 'period' in get_pipeline_metrics has no minimum/maximum; authentication enum in create_pipeline_webhook lacks description of when each type is valid). Forces LLM to guess valid formats. Violates pattern:constrained-input.
Destructive operations lack confirmation or dry-run patterns. approve_action, stop_pipeline_execution, and trigger_pipeline can halt deployments or consume resources, yet have no confirmation step, no destructiveHint in tool annotations, and no warning in description. Violates pattern:confirmation-request.
List tools (list_pipelines, list_pipeline_executions, get_pipeline_execution_logs) lack pagination. No limit, offset, or cursor parameters, and no total count in response. If a project has 1000+ pipelines or executions, response could exceed context window. Violates pattern:paginated-result.
No error recovery guidance. Tool descriptions and schemas do not indicate which errors are retryable, which require user intervention, or what the LLM should do next. Example: approve_action might fail if token is expired, should LLM retry? Fetch a new token? Violates pattern:recovery-guide.
Chaining IDs missing or undocumented. trigger_pipeline should return executionId so downstream tools can call get_pipeline_state(executionId). get_pipeline_state should return stageName and actionName so approve_action can use them. Without documented response fields, LLM must make extra lookup calls. Violates mxe:include-chaining-ids.
Parameter descriptions are minimal or missing context. 'pipelineName' appears in 9 tools but is described generically as 'Name of the pipeline' (18 chars). Does it accept aliases? Partial matches? ARN format? Parameter descriptions should be 50+ chars and include format/constraint examples. Violates pattern:tool-description.
No tool annotations (destructiveHint, readOnlyHint, idempotentHint). LLM cannot distinguish safe read-only calls (get_pipeline_state) from dangerous writes (stop_pipeline_execution, trigger_pipeline) without reading and parsing descriptions. Tool annotations are a current MCP spec feature (2026-07-28) and should be used.
No idempotency guidance. Tools that create or modify resources (trigger_pipeline, approve_action, tag_pipeline_resource, create_pipeline_webhook) do not document whether retrying with the same inputs is safe (idempotent) or risks duplicates/side effects. LLMs retry on ambiguous failures, without clarity, duplicates ensue. Violates pattern:idempotent-operation.