Loading ToolBench
Verification for agents that work unattended. Checks whether another agent's claim about its own work is true, whether a routine came back, whether work landed, and whether the checks are armed at all.
Four tools with clear, domain-specific descriptions (50 - 150 chars each) and well-structured input schemas. Naming is verb-based and unambiguous (verify_claim, check_routine_ran, check_work_landed, check_guards_armed). All tools are read-only with explicit risk annotations. However, output schemas are not documented, LLMs cannot predict return structure. Parameter descriptions are present but minimal (e.g., 'Absolute path to the repository' lacks format/validation hints). No error handling guidance, no pagination, no recovery paths. Tools are narrowly focused (good composition) but lack the depth of description and output documentation expected in production tools.
Ask whether the guards are actually switched on in THIS checkout. A hook that is configured in a settings file but not present in the working tree fails open, silently, and is indistinguishable from protection. This is the check that catches a disarmed checker, and it is the one most verification tooling does not have — including, by definition, itself.
Ask whether a scheduled routine actually completed, rather than merely being scheduled. A job that is enabled, a job that ran and did nothing, and a job that started and hung all look identical from the outside. This reads completion stamps and reports the last time the named routine genuinely finished.
Ask whether an agent's work reached somewhere a human will actually find it, or is stranded on a branch nobody opens. Work that has not landed where the operator looks has not been delivered, however good it is. Checks the trunk you name — it does not assume 'main'.
Check whether an agent's report about its own work is true. Give it the agent's message; it finds the checkable claims (commits, pushes, clean trees, tests, files, PRs, URLs) and checks each against the repository, the remote and the test runner. Returns one of three verdicts per claim: verified, FALSE, or UNVERIFIABLE. The third matters: a checker that cannot see must not answer 'false', or it accuses honest agents whose work is safely on a remote it could not reach. Claims that are not mechanically checkable are skipped and counted, so a low check-count is itself the signal.
Output schemas not documented. LLMs cannot predict return structure (e.g., does verify_claim return {verdicts: [...]} or {claims: [...], results: [...]}?). This forces agents to guess and risks misinterpreting results.
Parameter descriptions lack format/validation hints. 'Absolute path to the repository' does not specify: must it exist? Relative paths rejected? Symlinks followed? This invites invalid input from LLMs.
No error handling guidance. If verify_claim cannot reach a remote, does it return UNVERIFIABLE or an error? If check_routine_ran finds no completion stamp, what is returned? LLMs need explicit recovery paths.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
verify_claim description mentions 'three verdicts' (verified, FALSE, UNVERIFIABLE) but does not document the output structure. How are multiple claims returned? Is it an array of {claim, verdict} objects?
test_cmd parameter in verify_claim is optional but no guidance on what happens if omitted when claims mention test results. Does the tool skip test checks or fail?