Loading ToolBench
Citation-integrity and provenance for research synthesis: a blinded human->AI->adjudication dual-rating workflow with decision-gated, undoable Zotero write-back (single-user, local-first; searches PubMed, OpenAlex, Semantic Scholar, Crossref).
CiteVahti demonstrates solid domain-specific design with 11 well-intentioned tools covering citation integrity workflows. Strengths: all tools have descriptions (194 - 450 chars, within baseline 34 - 392 range); input schemas present with type definitions; clear verb-noun naming (init, triage, check_paragraph, verify_claims). Weaknesses: output schemas are not documented in the provided source, LLMs cannot infer what fields to expect from responses; error handling guidance is absent (no recovery hints, no categorization of retryable vs fatal errors); parameter descriptions lack format/constraint details (e.g., 'root' is described as 'Optional project root directory' but no validation rules stated); no enum constraints on multi-valued parameters; security-critical tools (init, open_review_panel) lack permission gates or audit trail documentation. Tool composition is sound (each does one thing), but chaining support is unclear, responses don't document which IDs/references downstream tools need.
Check-a-paragraph: paste a snippet the user just wrote and report, per sentence, which claims they've already VETTED, which NEED ATTENTION (with why + next action), and which are NEW/untracked. Read-only, no AI — the everyday in-the-writing loop. Offer to add + review the new ones; lead them to the ones needing attention.
Check whether a newer CiteVahti is published on PyPI. Read-only and user-initiated — it makes ONE outbound call to pypi.org only when asked, sends no user data, and never installs anything. Pair it with `status` (which reports the RUNNING version): use this when the user wonders if they're up to date, or after they reinstall the .mcpb / desktop app and want to confirm the new build is live (Claude Desktop caches the old extension — if still stale, remove the old one, restart, and add the newest). Returns current, latest, update_available, and a plain-language message safe to relay verbatim.
Start here. Returns where THIS project is and the ONE next thing to do — a state-aware first-run/resume guide that speaks to every state, including a brand-new (uninitialized or empty) ledger, which `triage` cannot. Read-only. Use this as the entry point when a user is new or asks "how do I start / what do I do next?": present `next.label` as the single next action (create the ledger → paste a paragraph → review the flagged claims → export the report) and surface any soft `blockers` (e.g. Zotero write not set up) without blocking their progress.
Create the project ledger (`.citevahti/config.json`) if it doesn't exist, and optionally PIN the agent's model so AI-extracting tools (`propose_claim` / `propose_revision`) can run. Run this FIRST — every other tool needs the ledger. Idempotent. Pass `model_id` (your own model, e.g. `"claude-opus-4-8"`) to record provenance; the snapshot defaults to the model id when not given. Reports the resolved root + config path so it's clear WHERE the ledger lives (the server's bound root, not the caller's cwd).
Output schemas not documented. LLMs cannot infer response structure (fields, types, pagination). Tools like triage, check_paragraph, methods return complex objects but no schema is visible in source.
Error handling lacks recovery guidance. No tool documents what errors are retryable, what the LLM should do on failure, or how to self-correct. E.g., pubmed_search may fail on invalid query but provides no guidance.
Parameter descriptions lack format/constraint details. 'root' appears in 10 tools but is described only as 'Optional project root directory', no validation rules, no examples of valid paths, no guidance on absolute vs relative.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Methods statement: the submission-ready methods paragraph auto-filled with this ledger's real numbers, plus the PRISMA-style 'how the literature was found' disclosure (whether an LLM was in the discovery loop, and that it only proposed leads — humans made every screening and rating decision). Read-only. Offer this for systematic reviews and pre-submission; it is the paste-ready methods + AI-use disclosure text.
Which AI model to trust as a second opinion, from THIS project's own record of validated divergences (a model earns standing by catching things the human adopted, not by agreeing). Read-only. Offer this when a serious review wants to pick — or add — a second-rater model: it returns a ranking, the recommended model, and, if you pass `model_id` and that model is rating low, a suggested better-evidenced alternative. Stays silent on models without enough resolved divergences to judge yet.
Open the human's loopback rating panel (idempotent; loopback-only). Closes the no-terminal dead-end: on the desktop-extension install only the bare stdio server runs, so when the claim-test prompt reaches the rate-first step the agent must be able to bring the rating surface up for the human. Adds no rating power — the panel is where the HUMAN rates, blind.
Staged PubMed search. Exact query presented to the user before fetching; returns candidate citations with abstracts, PMID, DOI, and OA status.
Report the current CiteVahti version, connection status to external services (PubMed, OpenAlex, Semantic Scholar, Crossref, Zotero), write backend availability, and write target information.
Risk-first triage: the few claims worth attention NOW, worst-first, each with the reason + the next action. Read-only. The friendly entry point — present these instead of asking the human to review every claim ("review these 3, not all 84").
Run citation-integrity tests over the manuscript's claims (read-only 4-state report).
Security-critical tools lack permission gates and audit trail documentation. init (writes config), open_review_panel (opens UI), and pubmed_search (external API call) do not declare required permissions or log who called them.
Tool chaining support unclear. Responses do not document which IDs/references downstream tools need. E.g., triage returns flagged claims but does not specify if claim_id is included for use in subsequent tools.