Project Auditron exhibits pervasive definition quality issues. While the server has 26 tools across AWS, GCP, and Azure domains (a commendable breadth), nearly every tool suffers from incomplete or missing schemas, undersized descriptions, and vague parameter documentation. The AWS tools (check_s3_public_access through check_cloudtrail_enabled) are defined in auditron/services/aws_service.py but their schema details are not visible in the provided source. The three reporting tools (generateSOCTool, generateISOTool, generateComplianceTool) defined in auditron-app/src/app/api/chat/route.ts have some parameter structure (organizationName, auditPeriod, controls, findings) but lack proper JSON Schema formalization and depth. Descriptions are universally terse (10 - 35 characters), failing to explain WHEN to use each tool, WHAT assumptions it makes, or HOW to interpret results. No tool declares required vs optional parameters, no enum constraints are visible, no pagination guidance for list-like tools, and error handling is completely absent from the definitions. This configuration makes it impossible for LLMs to reliably select and invoke tools without additional discovery overhead.
Descriptions fatally undersized: 23 of 26 tools have descriptions under 40 characters, far below the 194-char baseline and minimum 50-char guidance for LLM comprehension. Examples: 'Checks that all S3 buckets block public access' (44 chars), 'Checks that all EBS volumes in the configured region have encryption enabled' (76 chars but still vague on remediation and dependencies).
Expand ALL tool descriptions from current 10 - 40 chars to 100 - 250 chars following the 194-char baseline. For each check_* tool, specify: WHAT it audits (e.g. 'Verifies S3 bucket public access settings'), WHEN to use it (e.g. 'Before deploying production infrastructure'), WHAT it returns (pass/fail + details), and any prerequisites (e.g. 'Requires AWS credentials with s3:GetBucketPolicy permission').
Move credentials from tool parameters to server-side injection. Implement an authenticated endpoint that retrieves the current user's stored AWS/GCP/Azure credentials from Supabase (already integrated; see supabase/server.ts). Remove aws_credentials, gcp_credentials, azure_credentials parameters. Tools should infer credentials from the authenticated request context.
Formalize and document full JSON Schemas for all tool inputs and outputs. For check_* tools, declare output schema with fields like { status: 'pass'|'fail'|'error', summary: string, details: object, resource_count: number }. For reporting tools, declare output schema with { report_html: string, report_url?: string, generation_timestamp: string }.
Add enumerated constraints where applicable. For example, generateSOCTool's reportType parameter should accept only ['SOC2TypeII', 'SOC2TypeI'] (if those are the valid options), not freeform strings.
Implement error handling guidance in tool definitions and responses. Define which errors are retryable (credential expiry, rate limits) vs. permanent (permission denied, resource not found). Return actionable messages: 'S3 bucket not found. Check the bucket name and region. Try check_s3_public_access for a different bucket.' rather than raw API errors.
Score history
Overall score trend
↑ 15 points across a rubric change (v1 → v2)
39/100
Scored
Grade
Overall
Spec posture
Rubric
2026-09-22
F
39
2026-07-28+
v2
2026-03-09
F
24
-
v1
read onlyauthsource verified35/100
Checks for publicly accessible Azure Blob Storage containers.
Input schemas incomplete or inferred: 23 AWS/GCP/Azure check_* tools declare only a single optional aws_credentials/gcp_credentials/azure_credentials parameter object, but underlying JSON Schema is not visible in source. The schema for these credential objects is not documented, no properties, required fields, or type constraints are declared. This makes it impossible for LLMs to understand what fields are needed (access_key_id, secret_access_key, region?) or whether they are truly optional.
Credentials exposed as tool parameters: All AWS/GCP/Azure check_* tools accept aws_credentials, gcp_credentials, or azure_credentials as input parameters. Credentials must NEVER be passed as parameters, they appear in agent execution logs, traces, and prompt history, leaking secrets. Credentials should be injected server-side via environment variables or a secure vault, and the tools should infer which credential to use from the caller's context or a stored configuration.
No output schemas documented: Tools like generateSOCTool, generateISOTool, and generateComplianceTool have descriptions ('Generates a SOC 2 Type II compliance report in HTML format...') but no declared output schema. LLMs cannot know what fields, types, or structure to expect from the response, making it impossible to chain results into downstream tools or extract specific findings for the user.
No error handling guidance: None of the 26 tools include error recovery instructions. If a check_* tool fails (e.g., 'unable to authenticate with AWS'), the agent has no guidance on whether to retry, refresh credentials, or skip the control. Responses must tell LLMs: can I retry? Should I ask the user? Or is this unrecoverable?
Ambiguous tool naming for overlapping functionality: Multiple check_* tools perform security compliance checks (check_s3_public_access, check_rds_public_access, check_ebs_snapshot_public all check for public exposure). Naming does not distinguish the resource type clearly enough for LLMs to disambiguate intent, LLMs may invoke the wrong tool or waste reasoning cycles. Recommend prefixing by resource: check_s3_bucket_public_access, check_rds_instance_public_access, etc.
Parameter descriptions missing or vague: All 23 check_* tools list aws_credentials/gcp_credentials/azure_credentials as a parameter but describe it generically ('AWS credentials object containing access_key_id, secret_access_key, and region'). This is an example description only, it should not appear in the tool definition itself, and the LLM will likely treat it as an example value. The description should clarify whether the agent should provide credentials in full or whether the tool infers them from context.
Reporting tools lack input validation constraints: generateSOCTool, generateISOTool, and generateComplianceTool accept arrays (controls, findings, requirements) but do not specify minimum/maximum array length, required vs optional fields, or formats for nested objects. LLMs may pass empty arrays, malformed objects, or excessive data without guidance.
Disambiguate overlapping tool names. Prefix resource type: check_s3_bucket_public_access, check_rds_database_public_access, check_ebs_volume_encryption, etc. This reduces LLM confusion and makes tool discovery faster.
Document required vs. optional parameters explicitly. Mark aws_credentials, gcp_credentials, azure_credentials as truly optional ONLY if the tool can infer them from context; otherwise mark them required and explain how to obtain them.
For reporting tools (generateSOCTool, generateISOTool, generateComplianceTool), document the expected structure of 'controls', 'findings', 'requirements' array elements. Provide an example or schema snippet in the description.
Add scope/permission declarations to each tool. E.g., 'Requires: read:s3, read:iam' so the agent can be configured with minimum necessary permissions and audit trails are clear.
Implement pagination/limits for any check_* tools that could return large result sets. Cap results at 50 items by default and provide a limit parameter.