Scoring was not performed
Output schemas are completely undocumented. No tool specifies what fields will be returned (e.g., what does ask() return besides 'ids' and 'pending'?). LLMs cannot plan downstream actions or extract values when response structures are unknown.
Error handling is absent. No tool descriptions mention what errors can occur, what they mean, or how to recover. For example, if 'ask' fails due to pipe disconnection, does it retry? Fail immediately? Return partial results? No guidance.
Tool descriptions lack WHEN-to-use guidance. 'get_answers' vs 'wait_for_answers' both return answers, but descriptions don't clearly explain the tradeoff (polling vs blocking). An LLM must reason this out rather than having it explicit.
The 'instant_answers' feature is mentioned in 'ask' description but not formally part of any schema. Responses return 'instant_answers' field but this is not documented as part of the output contract, causing unpredictable behavior for LLMs.
'dismiss_questions' has sparse parameter documentation. The 'reason' parameter description is just 'Reason for dismissal', no indication of format, constraints, or impact. Is reason required? Any length limits?
'list_questions' status enum says 'expired' but 'ask' description and other contexts mention 'dismissed', naming inconsistency between what tools return and what they accept. This confuses LLMs about valid filter values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 25 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 57 | - | v1 |