Structured experimentation for iterative creative workflows. Provides tools for managing hypothesis-driven experiments with variable isolation and human-in-the-loop evaluation.
hunch_kit has 5 tools with clear naming (verb_noun pattern) and reasonable descriptions (100-200 chars). However, parameter descriptions are sparse or missing entirely. Input schemas are visible but lack detail: many parameters have no type hints in the schema (e.g., 'dimensions' in create_rubric is list[dict[str, Any]], which is too generic). Output schemas are completely undocumented, tools return plain strings with no structured format. Error handling is minimal: most tools return error strings rather than actionable recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all 5 tools being WRITE operations. The codebase shows validation logic (manifest.validate(), rubric.validate()) but errors are not surfaced to the LLM in a way that guides correction.
Create a new evaluation rubric from structured dimensions.
Create a new experiment directory and manifest.
Execute an experiment through its configured provider.
Record human evaluation scores for an experiment.
Add a child experiment to a parent's lineage.
Output schemas completely undocumented. All 5 tools return plain strings with no structured format. LLMs cannot parse results or chain downstream calls. Example: run_experiment returns 'Experiment: {id}\nStatus: {status}\n...', no JSON, no typed fields.
Parameter descriptions missing or too generic. 'dimensions' in create_rubric is list[dict[str, Any]] with no field-level documentation. 'scores' in score_experiment is dict[str, float] with no guidance on valid dimension names or score ranges. LLMs cannot validate inputs before calling.
No tool annotations despite all 5 tools being destructive (WRITE operations). Missing destructiveHint and idempotentHint annotations. Agents cannot reason about retry safety or side effects.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | <=2025-11-25 | v2 |
Error handling returns plain strings instead of actionable recovery guidance. Example: 'Error: Experiment directory already exists' does not suggest next steps. No error categorization (retryable vs user-fixable vs fatal).
Parameter 'provider' in init_experiment and run_experiment lacks enum constraint. Accepts free-form strings with default 'echo' but no documentation of valid providers. LLMs may hallucinate invalid provider names.