Loading ToolBench
Deterministic research integrity forensics system for detecting AI-generated content, numeric inconsistencies, statistical errors, and presentation anomalies in academic papers
Anti-Autoresearch defines 10 tools with explicit schemas and descriptions. Naming is domain-specific but mostly clear (build_*, check_*, adjudicate_*, attach_*, corrupt, run_eval). Descriptions are substantive (100 - 300 chars) and explain purpose well. However, several tools lack parameter descriptions entirely, and output schemas are not documented. Error handling is minimal, tools assume valid input and do not guide recovery. The server is READ_ONLY except for attach_refutation and corrupt, which is appropriate for a research integrity audit system, but this limits composability.
The deterministic reporter. Applies structural defenses against LLM-graded findings: ANCHOR rule (above-info findings must be span-anchored to ledger), CRITICAL SCRUTINY rule (critical findings must declare alternative explanations). Computes integrity verdict from weight-1 findings only, forcing zero-weight findings (AIS, advisory memos) to info severity.
The ONLY sanctioned way to write a refutation back into a findings file. Atomically attaches refutation results to a single finding by finding_id, verifies counter-evidence spans against ledger, and prevents finding mutation.
Deterministic evidence-ledger extractor. Turns a paper's source into schemas/claims.json: a span-anchored list of every checkable claim (numbers, comparisons, scope language, citations, captions, table cells). LaTeX-first (stable spans + line numbers); a plain-text path exists for PDF-extracted text at lower confidence.
Deterministic artifact inventory and observability-level derivation. Scans a paper directory, records which artifacts are present (with hashes), and derives the observability level by fixed rule: L2 (repo+results), L1 (latex only), L0 (pdf/text only).
Deterministic arm of the AI writing-style impression track (AIS). Emits zero-weight style impressions that do not affect integrity verdict. Implements AIS-DEFENSIVE-HEDGE: a conservative density screen for defensive 'we do not claim / not X but rather Y' writing patterns.
Output schemas not documented. Tools emit JSON files (claims.json, findings.json, report.json) but LLMs cannot see the structure of these outputs. This forces agents to guess field names and types.
Parameter descriptions missing or minimal. 'observability_level' (integer 0-2) lacks explanation of what each level means. 'out' paths are described generically without guidance on file format or structure.
No error handling or recovery guidance. Tools assume valid input (valid file paths, well-formed JSON). If a ledger file is malformed or missing, tools fail silently or with stack traces rather than actionable error messages.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Deterministic, arithmetic-only consistency checks. Emits findings.json that the adjudicator consumes. Checks: HP-DELTA-ERROR (stated relative/absolute improvement contradicts arithmetic), HP-NUM-INFLATE (metric-bearing headline %-number does not appear in any extracted table cell).
Presentation-signal checks for research integrity forensics. Detects presentation anomalies and surface-level integrity signals.
Statistical consistency checks for research integrity forensics. Detects statistical impossibilities and inconsistencies using GRIM, GRIMMER, Bhatia-Davis, and statcheck methods.
Inject a KNOWN integrity defect into a clean fixture. Each corruption is a list of (find, replace) edits. Every find MUST be present in the source (asserted) so a corruption can never silently no-op. Used by the eval harness to test deterministic checkers.
The calibration/regression harness. Proves the DETERMINISTIC core end-to-end with zero language model in the loop. For each fixture asserts: every required_patterns entry is present above info-severity (no false negatives), forbidden_above_info fixtures raise nothing above info (no false positives), adjudicated verdict matches expected_verdict/min_verdict. Reports precision/recall-style summary.
run_eval has no input schema (empty object). This tool is self-contained and requires no parameters, but the schema should document that explicitly (e.g., 'type: object, properties: {}, required: []').
attach_refutation's 'refuted' enum is string-valued ('true'/'false') rather than boolean. This invites type confusion, LLMs may pass boolean true/false instead of string 'true'/'false'.