A FastAPI-based server that searches AWS official documentation and retrieves relevant information about AWS services
Single tool with critical definition gaps. Tool name 'query_aws_docs' lacks action clarity (query vs search ambiguity). Description is present but generic (72 chars, below 100-char minimum for clarity). Input schema visible but parameters lack descriptions entirely, 'query' and 'service' have no guidance on format, constraints, or when to use each. No output schema documented. No error handling guidance. No pagination despite AWS docs being large. Violates 8+ core patterns from the rubric.
Searches AWS official documentation and retrieves relevant information about AWS services
Tool name ambiguity: 'query_aws_docs' vs 'search_aws_docs'. LLMs conflate query/search. Use 'search_aws_docs' to match the internal function name and clarify intent.
Input parameters lack descriptions. 'query' and 'service' have no guidance on format, constraints, or examples. LLMs cannot infer whether 'service' accepts 'ec2', 'EC2', 'Amazon EC2', or a full ARN.
No output schema documented. Response model exists (Response class with 'response' and 'source_url' fields) but is not exposed in tool definition. LLMs cannot plan downstream actions or extract structured data.
No pagination support. AWS docs searches can return hundreds of results; truncating to first result silently loses data. No limit parameter, no next_cursor, no total count. Violates paginated-result pattern.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 28 | 2026-07-28+ | v2 |
Error handling returns raw exception strings ('detail': str(e)). No recovery guidance, no categorization (retryable vs fatal), no actionable next steps. LLM receives 'Connection timeout' with no hint to retry or try a different service.