Flaky-test detector from CI history with quarantine suggestions
Single tool 'flakefinder_scan' has a basic description (67 chars) but critically lacks parameter descriptions. The input schema defines 'target' as a string with a description, but the tool itself provides no guidance on expected file formats, error handling, or output structure. No documented return schema. The description is generic and does not explain WHEN to use this tool or what 'quarantine suggestions' means. No error recovery guidance. Naming is acceptable (verb_noun form) but the tool lacks the depth expected for production use.
Flaky-test detector from CI history with quarantine suggestions. Returns JSON findings.
Parameter 'target' has a description in the schema, but the tool description does not explain expected input formats (JSON, JSONL, CSV, directory), file encoding, or size limits. LLMs cannot infer valid inputs.
No documented output schema. The tool returns JSON findings but LLMs do not know the structure of the response (field names, types, nesting). This prevents downstream tool chaining and forces LLMs to guess.
Tool description (67 chars) is too brief and lacks context. Does not explain WHAT flaky tests are, WHEN to call this tool, or what 'quarantine suggestions' means. Baseline for A+ tools is 50-200 chars with explicit purpose and use case.
No error handling guidance. Tool may fail on invalid file paths, unsupported formats, or malformed data, but the description provides no recovery hints (e.g., 'If file not found, check the path and try again').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | 2026-07-28+ | v2 |
Parameter 'target' accepts both files and directories but does not specify constraints: file size limits, directory depth, supported encodings, or number of files. LLMs may pass invalid inputs.