The server defines 3 tools with basic descriptions and schemas, but significant quality gaps reduce confidence. Tool names follow verb_noun convention (analyze_failure, get_results, save_results), which is positive. However, parameter descriptions are minimal or absent, input/output schemas lack depth, and error handling guidance is missing. The failure_mcp/server.py module shows tool registration via a static TOOLS list and a call_tool router, but the actual schema definitions are not visible in the provided source, they are referenced as being defined in failure_mcp.tools.analyze_test_failure, which is not included. This means schema assessment is constrained to what can be inferred. The single visible input schema (for analyze_failure with failure_context string, and get_results with min_confidence number) shows minimal type information and no validation constraints. Descriptions like 'Send a test failure to the LLM for diagnosis' are brief (7-10 words) and lack context about prerequisites, when to use the tool, or what dependencies exist. No output schemas are documented, no error recovery guidance is provided, and no tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite destructive semantics on save_results.
Tools (3)
analyze_failureread onlyauthsource verified45/100
Send a test failure to the LLM for diagnosis
get_resultsread onlysource verified47/100
Return all diagnosis results filtered by minimum confidence
save_resultswritesource verified43/100
Flush all accumulated results to failure_analysis.json
Output schemas are not documented for any tool. The LLM cannot plan downstream tool calls or extract required fields without knowing what each tool returns.
Parameter descriptions lack validation constraints: no min/max for numeric types, no format specs for strings, no enum values for enumerable fields. This forces LLMs to guess at valid ranges and formats, increasing error rates.
save_results tool description does not declare that this is a destructive/state-modifying operation. Per pattern:command-tool, descriptions of mutating tools MUST explicitly state what state changes occur. No confirmation or dry-run capability is offered.
save_results
Recommendations
Document complete input AND output schemas for all tools. Example for analyze_failure: input {failure_context: {type: string, description: '...', minLength: 10, maxLength: 5000}}, output {analysis: {type: string}, confidence: {type: number, minimum: 0, maximum: 1}, recommendations: {type: array}}.
Expand tool descriptions to 100 - 200 characters and include: WHAT the tool does, WHEN/WHY to call it, any prerequisites, and what fields it returns. Example: 'Diagnose a single test failure using AI. Pass the full failure output (assertion error, stack trace). Returns a JSON object with analysis text, confidence score (0 - 1), and a list of recommended fixes. Call this before get_results to populate the results cache.'
Add validation constraints to parameters: min/max for min_confidence (e.g., minimum: 0, maximum: 1), format specs for strings (e.g., minLength, maxLength, pattern), enums for choice fields. Include these in descriptions: 'min_confidence: Minimum confidence threshold (0 - 1, e.g. 0.75 to see only high-confidence results).'
Add tool annotations via the current MCP spec (2026-07-28): mark analyze_failure and get_results with readOnlyHint=true; mark save_results with destructiveHint=true to signal state changes.
Enhance save_results description to declare destructive semantics: 'Persist all accumulated AI diagnosis results to failure_analysis.json. WARNING: This overwrites any existing file. Returns {saved: number, path: string, timestamp: ISO8601}. Consider calling get_results first to preview results before flushing.'
Tool descriptions are too brief and lack context about prerequisites, follow-up steps, and error scenarios. Average description length across tools is ~56 characters; baseline is 194 chars. Descriptions do not guide LLM reasoning about when to invoke each tool or how to chain them.
No error handling guidance. Tools provide no recovery hints, categorization of errors (retryable vs. fatal), or enriched error messages. If analyze_failure fails, the LLM has no guidance on what to do next.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) are absent. The LLM cannot determine from tool metadata alone whether a call is safe to retry, will modify state, or has side effects. save_results is clearly destructive but carries no hint.
No pagination support on get_results. If results grow large, returning all at once will bloat the context window. Should accept limit and offset/cursor parameters and return a total count.
get_results
Add error handling guidance: If analyze_failure receives invalid failure_context, return error 'Invalid failure context: expected a non-empty string (10 - 5000 chars). Example: "AssertionError: Expected status 200, got 500..."'. Provide recovery hints: 'Use the test output or error message from your CI/CD run.'
Document what failure_context should contain: 'The complete test failure output, including test name, assertion error, and stack trace. Example: "test_api_response: AssertionError: expected status 200 but got 500. Stack: ..."'. This guides LLMs in extracting and formatting test output correctly.
Add a dry_run parameter to save_results (default false): When true, returns what WOULD be saved without modifying the file. This enables LLMs to preview before committing, reducing mistakes.
Document the relationship between tools: 'Call analyze_failure for each test failure you want diagnosed. Call get_results periodically to review findings with min_confidence filter. Call save_results at the end of your analysis session to persist the results.' This guides LLM composition.