MCP + RAG based intelligent automation testing platform with tools for knowledge retrieval, test case generation, API testing, and failure analysis
Server has 10 tools with basic descriptions and input schemas, but significant gaps in parameter documentation and output schema clarity. Most tool descriptions are present (10/10) but lack depth, averaging ~80 chars, well below the 194-char baseline for A+ tools. Parameter descriptions are sparse: many parameters lack detail on format, constraints, or valid ranges. No output schemas documented. Error handling guidance is absent. Tool naming is reasonable (verb_noun pattern) but some names are generic ('generate_test_cases', 'analyze_failure'). No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications in metadata. STDIO transport caps protocol readiness at 50, limiting overall score.
Analyze a test failure and provide root cause analysis and recommendations
Generate a structured bug report from test failure analysis
Generate test cases based on requirements and context retrieved from the knowledge base
Get a summary of a specific document from the knowledge base
Retrieve test execution report with results, metrics, and analysis
List all available collections in the knowledge base
Query failed test cases from the test history
Query the knowledge hub using hybrid search (BM25 + dense retrieval) to retrieve relevant test documentation and context
Output schemas not documented. Tools return results but LLMs cannot infer response structure, forcing guesswork on field names and types. Breaks tool chaining and wastes tokens on parsing.
Parameter descriptions lack format/constraint details. E.g., 'top_k' has no min/max bounds; 'collection' has no enum of valid collections; 'severity' in generate_bug_report has enum but other params do not. LLMs cannot validate inputs without explicit constraints.
No error handling guidance. Tools lack recovery hints (e.g., 'If collection not found, call list_collections() first'). Agents cannot self-correct on failures.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 42 | <=2025-11-25 | v2 |
Retrieve test context and requirements from the knowledge base for a given test scenario
Run API tests and return execution results
Tool annotations missing. Metadata declares risk classifications (READ_ONLY, WRITE) but tools lack readOnlyHint/destructiveHint/idempotentHint annotations in schema. Agents cannot infer safety properties from definitions alone.
Pagination not documented. Tools like query_knowledge_hub and query_failed_cases accept 'top_k'/'limit' but no cursor, offset, or total_count guidance. Large result sets risk context window exhaustion.