Deterministic compatibility gates for observable AI-agent behavior across model, prompt, memory, and tool changes — CLI, SDK, CI, and MCP.
Behaviorlock defines 5 tools with complete input schemas and descriptions. All tools follow verb_noun naming (validate, compare, explain, fingerprint, render). Descriptions are clear and domain-specific (avg ~100 chars), exceeding the 10-char minimum. All parameters have type definitions and descriptions. However, output schemas are not documented, LLMs cannot predict response structure. Error handling is minimal; tools return generic JSON without recovery guidance. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only. Parameter descriptions lack format/constraint details (e.g., 'Path to trace file' doesn't specify file extension, encoding, or validation rules). No pagination or result-limiting patterns despite tools that could return large datasets.
Compare a baseline and candidate trace against deterministic observable-behavior contracts.
Compare traces and return one assertion with selected baseline/candidate values and the exact reason.
Create a stable SHA-256 fingerprint of scenario status and observations, excluding run metadata.
Compare traces and render JSON, Markdown, standalone HTML, SARIF, or JUnit XML.
Validate a behavior contract and optional portable trace files without comparing them.
Output schemas not documented. LLMs cannot predict response structure or plan downstream tool calls. Tools return JSON via jsonResult() helper but no schema is declared.
No tool annotations despite all tools being read-only. Missing readOnlyHint in tool registration prevents LLMs from understanding safety properties and optimizing retry logic.
Parameter descriptions lack format constraints. 'Path to trace file' does not specify file extension (.json?), encoding (UTF-8?), or validation rules. LLMs cannot self-correct invalid paths.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
No error recovery guidance. Tools return generic JSON errors (e.g., 'not_found') without actionable next steps. LLMs cannot determine if error is retryable or requires user intervention.
Implicit contract file fallback via environment variable (BEHAVIORLOCK_CONTRACT) is undocumented in tool descriptions. LLMs cannot predict behavior when contract parameter is omitted.