DataHub schema-change rehearsal: query-level hard/soft/unknown breakage forecasts
Three tools with partial schemas and descriptions. rehearse_schema_change and explain_finding have reasonable parameter documentation but lack output schema documentation. write_back_forecast has minimal description (67 chars, below 100-char baseline). All tools return JSON strings rather than structured objects. No error handling guidance, no recovery paths, no input validation messaging. Parameter descriptions are present but generic (e.g., 'Dataset URN' lacks format/constraint details). No enum constraints on change_kind despite accepting only 'rename'/'drop'. Tool composition is reasonable (three distinct concerns), but output structure is undocumented for LLM planning.
Detail one finding by query_id for the same change as rehearse. Never on camera.
Rehearse a schema change: HARD/SOFT/UNKNOWN/CLEARED forecast + write_payload + repairs (HARD/SOFT patches; CLEARED/UNKNOWN refused). On-camera tool. adjudicate='binder' (default) uses classify-only. Host applies write_payload via DataHub MCP — do not recompute.
Gated write-back. Prefer DataHub MCP applying write_payload as-is.
Output schemas undocumented. All tools return JSON strings; LLMs cannot plan downstream calls without knowing response structure (fields, types, nesting). Baseline: 100% of A+ tools document return types.
change_kind parameter accepts only 'rename'/'drop' but is declared as free-form string with no enum constraint. Invites hallucinated values. Should declare enum: ['rename', 'drop'].
No error handling guidance. Tools lack recovery hints (e.g., 'If dataset not found, try search_datasets()' or 'Invalid change_kind: must be rename or drop'). Baseline: error responses must tell LLM what to do next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
write_back_forecast description is 67 characters (below 100-char minimum). Lacks WHEN to use it, prerequisites, or state-modification warning. Baseline: 10 - 1024 chars, avg 194 chars.
Parameter descriptions lack format/constraint details. 'Dataset URN to rehearse' does not specify URN format, valid platforms, or example structure. Baseline: descriptions should state format, range, allowed values.