Generates falsifiable worst-case fleet hypotheses, compiles each into the smallest bounded targeted-scan probe with a control cohort, and returns a support/falsification verdict that requires discrimination from the controls.
Three tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. All tools lack descriptions for individual parameters (e.g., 'assets' array has no description of what constitutes a valid asset object). Output schemas are not documented, LLMs cannot predict return structure. No error handling guidance. Tool names follow verb_noun pattern but descriptions are terse (40-80 chars, below the 50-200 char LLM-optimized baseline). No parameter constraints (enums, ranges, formats). Security-focused domain (hypothesis testing, probe planning) but no indication of permission gates or audit trails.
Compile one hypothesis into a bounded targeted probe plan with optional control cohort.
Evaluate target and control observations against a hypothesis and return a falsification/specificity verdict.
Generate deterministic evidence-seeking security hypotheses from fleet observations.
No parameter descriptions. 'assets', 'observations', 'hypothesis', 'plan', 'result' parameters lack any explanation of expected structure, format, or constraints. LLMs cannot infer what constitutes a valid asset object or observation.
Output schemas not documented. Tool descriptions do not specify what fields are returned or their types. LLMs cannot plan downstream calls or extract required data (e.g., does nightmare_generate return hypothesis IDs? What fields does a hypothesis object contain?).
No error handling guidance. Tools do not document failure modes, recovery steps, or actionable error messages. If hypothesis compilation fails due to insufficient budget, the LLM has no guidance on what to do next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2025-06-18+ | v2 |
Descriptions too terse (40-80 chars). Below LLM-optimized baseline of 50-200 chars. 'Generate deterministic evidence-seeking security hypotheses from fleet observations' lacks context on WHEN to call this tool, what prerequisites exist, or what the output enables.
No parameter constraints. 'budget' has minimum=1 but no maximum. 'options' object is unconstrained. No enums, patterns, or format declarations. LLMs can pass invalid values without guidance.