Loading ToolBench
Repository audit with four Claude lenses, an adversarial verifier and SARIF, JSON and Markdown output.
Black Heron exposes 3 tools with complete input schemas and reasonable descriptions. Tool names follow verb_noun convention (audit_repository, verify_findings, quick_scan). Descriptions are 100-200 chars, within baseline range. However, output schemas are NOT documented, LLMs cannot predict what fields to expect from responses. Parameter descriptions are present but lack detail on constraints, formats, and error recovery. No tool annotations (readOnlyHint/destructiveHint) despite audit_repository being a WRITE operation. Error handling is absent from tool definitions.
Run a full Black Heron audit on a local repository. 4 lenses (code_quality, governance, drift, blind_spot) + adversarial Opus verifier. Returns summary metrics; full report at out_dir.
Fast single-lens scan (code_quality only, no verifier). Cheaper and faster than full audit. Use for routine pre-commit checks.
Run the adversarial verifier (Opus 4.7) on findings produced by another tool. Accepts a JSON file with a 'findings' array matching the Black Heron Finding schema. Returns kept/rejected partition.
Output schemas not documented. Tool descriptions state what is returned (e.g., 'Returns summary metrics; full report at out_dir') but do not specify the JSON structure, field types, or nested objects. LLMs cannot plan downstream operations or extract specific fields without seeing the schema.
Missing tool annotations. audit_repository is marked Risk: WRITE but has no destructiveHint or idempotentHint in the schema. verify_findings and quick_scan are READ_ONLY but lack readOnlyHint. Tool annotations help LLMs reason about retry safety and side effects.
Parameter descriptions lack constraint details. 'cost_cap_usd' and 'time_cap_seconds' have no min/max bounds stated. 'lenses' accepts comma-separated values but does not enumerate valid lens names or explain the default behavior. 'repo_path' requires absolute path but does not state this explicitly.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
No error recovery guidance. Tool descriptions do not explain what errors are possible, how to interpret them, or what the LLM should do next. E.g., if repo_path is invalid, should the LLM retry, ask the user, or call a discovery tool?
Missing dependency hints. verify_findings accepts a findings_json_path but does not explain where this file comes from or that it must be produced by audit_repository or another compatible tool. This forces the LLM to infer the relationship.