Loading ToolBench
Read-only AWS security audit tools for AI agents, served over the Model Context Protocol
Strong foundation with clear, well-documented audit tools. All 12 tools have explicit descriptions (avg ~180 chars, within baseline 194), proper naming (verb_noun pattern: audit_*, account_*), and documented input schemas. Tool outputs follow a consistent envelope contract (check, ok, findings, scanned). However, output schemas are not formally documented in JSON Schema format, they are described in prose. Parameter descriptions are present but minimal (10-30 chars typical). Error handling is implicit (tools fail silently or return empty findings) rather than explicit. No tool annotations visible in code despite READ_ONLY being defined. Missing: per-parameter type constraints (enums, ranges), output schema documentation, error recovery guidance.
Cheap posture snapshot of the audited AWS account. Collects the account id (sts.GetCallerIdentity), iam.GetAccountSummary counters (Users, AccountMFAEnabled, AccountAccessKeysPresent), and the account-level S3 public access block (s3control.GetPublicAccessBlock). Findings: HIGH when the root account has no MFA (AccountMFAEnabled != 1); MEDIUM when the account-level S3 public access block is missing or any of its four flags is disabled.
Audit EBS volumes and snapshots for exposure risks. Scans every EBS volume in the region and flags unencrypted volumes as MEDIUM. Scans every snapshot owned by the account and checks its createVolumePermission attribute; a snapshot shared with the "all" group is public and is flagged CRITICAL.
Run every registered audit check with default arguments and aggregate the results into a single account posture report. Discovers all audit_*/account_* functions in the tool modules and runs each one; a failing check is recorded in `errors` and never aborts the rest. Returns the standard {check, ok, findings, scanned} envelope where findings is the concatenation of every sub-report's findings and scanned is the number of checks run, plus severity_counts (LOW/MEDIUM/HIGH/CRITICAL), posture_score, grade, checks_run, and errors. The score is weighted: each CRITICAL finding costs 10 points, HIGH 5, MEDIUM 2, LOW 1, and posture_score = max(0, 100 - total weight). Grades: A >= 90, B >= 75, C >= 60, D >= 40, else F.
Output schemas not formally documented. Tools return structured envelopes (check, ok, findings, scanned, severity_counts, posture_score, grade) but no JSON Schema definition is visible. LLMs cannot reliably parse or plan downstream operations without explicit output type documentation.
Tool annotations declared in common.py (READ_ONLY) but not applied to tool registrations. Code shows @mcp.tool(annotations=READ_ONLY) pattern is available but not visible in actual tool module registrations. Clients cannot infer read-only safety without explicit annotations.
Region parameter descriptions are minimal ('AWS region to query (defaults to AWS_REGION or us-east-1)') and lack constraint documentation. No enum of valid regions, no guidance on format (us-east-1 vs US-EAST-1), no error recovery if region is invalid.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
Audit Lambda function resource policies for overly broad invoke access. Scans every Lambda function in the region and inspects its resource policy. An Allow statement with a wildcard principal ("*" or {"AWS": "*"}) and no SourceArn/SourceAccount condition is HIGH: anyone can invoke the function. A service principal without any source condition is MEDIUM: any account's use of that service can invoke it.
Audit every S3 bucket in the account for public exposure. For each bucket this checks: the Public Access Block configuration (missing or any of the four flags disabled yields a MEDIUM finding), the bucket policy (any Allow statement with principal "*" or {"AWS": "*"} yields a HIGH finding), and the bucket ACL (grants to the AllUsers or AuthenticatedUsers groups yield a HIGH finding).
Audit RDS instance posture: public accessibility, storage encryption, and deletion protection. Scans every RDS DB instance in the region. A publicly accessible instance is a HIGH finding, unencrypted storage is MEDIUM, and disabled deletion protection is LOW.
Audit the root account for missing MFA and active root access keys. Reads the IAM account summary. No root MFA is a HIGH finding; any root access key present is a CRITICAL finding.
Audit IAM users for active access keys older than max_age_days. Scans every IAM user and flags each Active access key whose age exceeds max_age_days. Severity is HIGH when the key's owner has no MFA device (a leaked key is the only factor), MEDIUM when the owner has MFA.
Audit CloudTrail trail posture in the account. Checks per trail: trail exists at all (zero trails is a CRITICAL finding on its own), the trail is actively logging (GetTrailStatus.IsLogging) CRITICAL if not, IsMultiRegionTrail MEDIUM if single-region, LogFileValidationEnabled MEDIUM if disabled, KmsKeyId present LOW if logs are not encrypted with a customer managed key.
Audit IAM users that can log in to the console without MFA. Scans every IAM user, checks which ones have a console login profile, and flags those with no MFA device as HIGH severity. Users without console access are counted but never flagged.
Audit EC2 security groups for ingress rules open to the world. Scans every security group in the region and flags each ingress rule whose source is 0.0.0.0/0 or ::/0. Rules allowing all traffic (protocol -1) or covering a sensitive port (SSH 22, RDP 3389, MySQL 3306, PostgreSQL 5432, Redis 6379, Elasticsearch 9200, MongoDB 27017) are HIGH; any other world-open port (for example 80 or 443) is MEDIUM.
Example audit that scans nothing and returns an empty findings envelope. Demonstrates the contract every tool follows: a report() envelope with check, ok, findings and scanned, so agents can trust an empty result.
Error handling is implicit (tools return empty findings on failure) rather than explicit. No error messages, recovery guidance, or categorization (retryable vs fatal). Agents cannot distinguish 'no findings' from 'API call failed'.
max_age_days parameter in audit_stale_access_keys lacks numeric constraints. No min/max bounds documented; LLMs could pass negative, zero, or absurdly large values (e.g. 999999 days). Should specify 1 - 365 range.