Loading ToolBench
An MCP connector that verifies a claim's grounding against reality — no LLM in the verification path.
Groundcheck has well-structured tool definitions with clear, domain-specific descriptions (avg 180 chars) and complete input schemas. All 5 tools are explicitly registered with @mcp.tool() decorators in server.py. Naming follows verb_noun pattern (check_quote, check_citation, check_code, check_repo, check_math). However, output schemas are not documented, callers must infer the {status, method, evidence, detail} structure from docstrings rather than formal schema. Parameter descriptions are present but minimal (avg 40 chars). No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles (check_code is DESTRUCTIVE, others READ_ONLY). Error handling guidance is implicit in descriptions but not formalized.
Verify that an arXiv id or DOI actually resolves, and return its real title. Pass an arXiv id (e.g. '2412.18004' or 'arXiv:2412.18004') or a DOI. A 'checked' verdict includes the genuine title, so a plausible-looking but fabricated identifier is caught as 'refuted'.
Run a self-contained Python snippet and confirm its stdout contains `expected_output`. WARNING: this executes the code. Only pass code you would run yourself; it is not sandboxed from the network or filesystem. Returns refuted if the snippet errors or prints something other than claimed.
Confirm an arithmetic expression equals a claimed result. Evaluates `expression` (numeric literals and + - * / // % ** only) and compares to `claimed_result`. Catches arithmetic a model asserted without computing.
Verify that an exact quote appears on a web page. Fetches `url` and confirms `quote` is present in its visible text. Use before attributing a statement to a source. Returns refuted if the page loads but the quote is absent -- i.e. the attribution is fabricated.
Confirm that a string (or regex) actually occurs in files under a local path. Use to verify a claim about what a codebase or document contains -- 'the README reports 0.9086', 'the function is defined in src/' -- rather than trusting the assertion. Returns the real matching lines.
Output schema not formally documented. Callers must infer {status, method, evidence, detail} structure from docstrings. LLMs cannot plan downstream operations without knowing return fields.
Tool annotations missing. check_code is DESTRUCTIVE (executes arbitrary Python) but lacks destructiveHint. Others are READ_ONLY but lack readOnlyHint. Agents cannot reason about safety without explicit annotations.
Parameter descriptions are minimal (avg 40 chars). 'The exact quote to verify' and 'A self-contained Python code snippet to execute' lack format/constraint guidance. LLMs cannot infer edge cases (multiline quotes, code with imports, regex patterns).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Error recovery guidance not formalized. Docstrings mention 'refuted' and 'unverifiable' verdicts but do not explain when each occurs or what the LLM should do next. No structured error classification (retryable, user-fixable, fatal).
check_code accepts arbitrary Python without sandboxing. Docstring warns 'not sandboxed from network or filesystem' but no input validation, timeout, or resource limits are visible. Agents could hang or corrupt state.