Loading ToolBench
Verifier-grounded evaluation tools for AI systems: exposure quarantine, paired promotion gates with exact McNemar statistics, leakage audits, bank health diagnostics, SafePatch conservation edits, graph counterexample search, memory relevance scoring, and agent event replay.
Whetstone defines 19 tools with explicit schemas and descriptions visible in source. Naming is domain-specific but mostly verb-forward (inspect_, audit_, gate_, bank_, safe_, hunt_, memory_, replay_, report_card_, open_bench_). Descriptions range 50 - 200 chars, meeting baseline. However, parameter descriptions are sparse or missing for complex nested objects (e.g., audit_leakage's 'fingerprint_max_n' constraint '3 to 5' is stated but not in schema; gate_results 'policy' object lacks nested field descriptions). Output schemas are not documented. Error handling is implicit (ProductInputError class exists but recovery guidance is minimal). Tool composition is sound, each tool has one responsibility. Security: no credentials exposed as params. Overall: solid domain coverage with good naming, but incomplete parameter documentation and missing output schema specs hold it back from 70+.
Find compromised evaluations through three evidence tiers: exact identity matching, behavioral fingerprinting on graph-DSL expressions, and text similarity scoring.
Diagnose the exam bank: measure item flakiness, map the frontier of system performance, and queue actions based on repeated outcomes.
Catalog of the eight stateless verifier tools with names, descriptions, and input schemas.
A complete, valid request payload for every tool.
PASS / HOLD / BLOCK verdict using paired exact McNemar statistics. Baseline and candidate must cover identical item IDs; a higher aggregate score cannot erase a regression.
Service health, version, and report-card readiness.
Output schemas not documented. Tools return results but LLMs cannot plan downstream calls without knowing field names and types. E.g., inspect_promotion, audit_leakage, gate_results lack documented return structures.
Nested object parameters lack field-level descriptions. gate_results 'policy' object has 'confidence_alpha' but no description of what it controls or valid range. audit_leakage 'ns' object is undocumented.
Error recovery guidance missing. ProductInputError class exists but tool descriptions do not explain what to do if validation fails or how to correct input. E.g., 'similarity_threshold must be in [0.5, 1.0]' is stated but no guidance on what happens if violated.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Attack a graph conjecture: bounded exact witness search for graph-DSL expressions. Returns a counterexample graph if found within the search budget.
Input schema definitions for all stateless tools.
Full promotion decision: requires a complete post-quarantine cohort. Pairs exam, exposure, baseline, and candidate results to issue a verdict.
Catch attention hijacks: rank memories by relevance to an objective, conditioned on goal entities and token budget.
Opt-in, self-attested public Open Promotion Bench receipts.
One sanitized public benchmark receipt.
Start a paired six-item scope-integrity cohort for Open Promotion Bench baseline and candidate evaluation.
Grade paired baseline and candidate tasks and optionally publish a sanitized self-attested receipt to Open Promotion Bench.
Event flight recorder: reconstruct explicit emulator events, branches, and rewinds from an event log or emulator result.
Start a disposable checker-graded report card: issues a one-shot 6-item graph-repair exam session.
Grade a report card and issue a signed receipt: evaluates responses against checker spec and destroys the one-shot session.
Patch without collateral damage: apply caller-specified Markdown edits and verify conservation of protected tokens and sections.
Aggregate, privacy-preserving usage counters.
Metadata tools (catalog, examples, input_schemas) have minimal descriptions and no clear use case. LLMs may not understand when to call them vs. reading documentation directly.
Stateful session tools (report_card_start/submit, open_bench_start/submit) require session_id correlation. No documented guidance on session lifecycle, timeout, or what happens if session_id is invalid or expired.