Loading ToolBench
A FastAPI-based MCP gateway that provides a unified interface to AWS services including EC2, EKS, Cost Explorer, and AWS Knowledge Base. Supports credential management, rate limiting, and multiple AWS service integrations.
AWS MCP Gateway exposes 16 tools across 4 service modules (EC2, EKS, Cost Explorer, AWS Knowledge). Tool definitions are present with schemas and descriptions, but several quality gaps emerge: (1) Most descriptions are generic and under-optimized for LLM selection (e.g., 'List EC2 instances in the specified account' lacks context on WHEN to use vs alternatives). (2) Parameter descriptions exist but are minimal (e.g., 'Optional filters for EC2 instances' doesn't specify filter format, valid keys, or structure). (3) Output schemas are not documented in tool definitions, callers don't know what fields to expect. (4) Error handling lacks recovery guidance, no actionable error messages visible in code. (5) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels (READ_ONLY vs WRITE). (6) Pagination parameters missing from list tools (list_instances, list_clusters, list_nodegroups) despite baseline pattern requiring them. The code is functional and organized, but falls short of production-grade tool design.
Get details of an EKS cluster
Get details of a specific EC2 instance
Get pod metrics for an EKS cluster
Get cost and usage data for the account
Get cost forecast for the account
Get available values for a specific cost dimension
Retrieve specific AWS documentation page
Output schemas not documented. Tool definitions specify input parameters but omit return type schemas. Callers and LLMs cannot know what fields to expect, forcing them to infer structure from examples or trial-and-error.
Pagination parameters missing from list tools. list_instances, list_clusters, list_nodegroups return potentially large result sets but lack limit, offset, page_size, or next_cursor parameters. Without pagination, context window overflow is inevitable.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | <=2025-11-25 | v2 |
Get Savings Plans utilization data
Get main documentation page for an AWS service
Get AWS What's New announcements
List EKS clusters in the specified account
List EC2 instances in the specified account
List node groups for an EKS cluster
Search AWS documentation
Start an EC2 instance
Stop an EC2 instance
Descriptions are generic and lack LLM-optimized guidance. Examples: 'List EC2 instances in the specified account' (44 chars) does not explain WHEN to call this vs describe_instance, or what filters are valid. 'Get details of a specific EC2 instance' (38 chars) is vague on whether it returns tags, metrics, or just basic config.
Parameter descriptions are minimal and lack format/constraint details. Example: 'filters' parameter in list_instances is described as 'Optional filters for EC2 instances' with no indication of valid filter keys (e.g., instance-state-name, tag:Name) or structure (flat dict vs nested). Similar issues in get_cost_and_usage 'metrics' array and 'group_by' array, valid values not enumerated.
No tool annotations despite clear risk levels. Tools are marked with Risk: READ_ONLY or Risk: WRITE in documentation, but no readOnlyHint, destructiveHint, or idempotentHint annotations are visible in tool definitions. LLMs cannot distinguish safe (retry-able) calls from irreversible ones (start_instance, stop_instance) without explicit hints.
No error recovery guidance in tool definitions or visible error handling code. HTTP execute_mcp_tool endpoint returns generic HTTPException(500, detail=str(e)) with stack trace, no structured error classification, retry hints, or user-actionable guidance. Example: 'Invalid AWS Account ID' vs 'Account not found. Try list_accounts() to see valid IDs.'
Credentials exposed in environment: scripts/run-with-credentials.sh exports AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY directly. No mention of server-side secret injection via Vault, SSM Parameter Store, or IAM role assumption. If credentials leak into logs or traces, damage is severe.
No result limits documented for tools that query external data. search_documentation accepts limit parameter (default 10, max 50), good. But get_whats_new and get_cost_and_usage lack explicit result-size caps. LLMs cannot know if 500 results are possible, risking context overflow.
account_id parameter universally required but no validation shown. Tools like start_instance require account_id parameter, but no tool validates format (12-digit AWS Account ID) or checks if the account exists. Invalid input should fail fast with 'Invalid account ID format' not 500 error.
No idempotency guarantees documented. start_instance and stop_instance lack explicit idempotency statements. If called twice, does the second call succeed (idempotent) or fail with 'already started'? LLMs need this to plan retries safely.