Artifact-aware framework for building self-evolving AI agent harnesses.
Harneloop exposes 26 tools with consistent naming (verb_noun pattern) and descriptions present for all tools. However, critical gaps undermine quality: (1) Input schemas lack type information beyond basic type hints (Path, string, list[str], boolean), no JSON Schema with constraints, enums, or ranges; (2) Parameter descriptions are minimal (10-30 chars) and lack actionable guidance on format, constraints, or dependencies; (3) No output schemas documented, LLMs cannot predict return structure; (4) Error handling is absent from tool definitions; (5) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite 15+ WRITE operations. Tool names are clear (init_unit, create_attempt_plan, promote_candidate) but descriptions are terse. Example: 'create_attempt_plan' has description 'Create an agent-authored attempt plan for testing a harness' (67 chars) but parameters like 'goal', 'method', 'action' lack format guidance. No evidence of pagination, batch operations, or composition patterns.
Record user confirmation or delegation of intake context
Add an artifact to a run
Add an observation to an attempt plan
Add evidence to support a candidate
Evaluate an attempt and choose its next lifecycle action
Create an environment contract for a harness unit
Create an agent-authored attempt plan for testing a harness
No JSON Schema with constraints, enums, or ranges. Parameters have Python type hints (Path, string, list[str]) but no formal JSON Schema validation, min/max bounds, or enum constraints. LLMs cannot validate input before calling.
Parameter descriptions are minimal (10-30 chars) and lack actionable format guidance. Example: 'template' param in init_unit has description 'Template to use for unit creation' (33 chars) with no mention of valid values (blank, etc.) or format.
No output schemas documented. Tool definitions show input parameters but no return type or field structure. LLMs cannot predict what fields to extract or chain to downstream tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 40 | 2026-07-28+ | v2 |
Create a candidate patch workspace for harness improvements
Export a harness unit for use with external adapters
Mark a run as finished
Create a new harness unit
List all candidates in a harness unit
List all registered harness units
Package a harness unit for distribution
Promote a candidate to become the new harness version
Rebase a candidate against the current base version
Register an existing harness unit
Remove a harness unit from the local registry
Record one onboarding context field
Rollback a harness unit to a previous version
Run diagnostic checks on the Harneloop environment
Update the status of a candidate
Set the target task brief for a harness unit
Start a new run for testing a harness
Add missing current protocol files to an existing unit
Validate a harness unit for correctness
15 WRITE tools (init_unit, create_attempt_plan, promote_candidate, etc.) lack destructiveHint or idempotentHint annotations. LLMs cannot determine which calls are safe to retry or have irreversible consequences.
No error handling guidance in tool definitions. No recovery hints, retryability classification, or actionable error messages. Agents cannot self-correct on failures.