Loading ToolBench
Read-only Model Context Protocol server over committed evidence from a physics-conditioned SatNOGS review queue. Exposes tools for querying observations, checking claims against evidence packets, and accessing gate verdicts.
Server has 7 tools with complete descriptions and input schemas visible in .bob/mcp.json and apps/web/public/data/mcp.json. Naming follows verb_noun convention (queue_top, queue_size, observation, check_claim, gate_status, receipt, run_acceptance). Descriptions are substantive (100-250 chars) and explain WHAT and WHEN. However, parameter descriptions lack detail on constraints, formats, and error recovery. Output schemas are not documented. Error handling guidance is absent. The server is read-only except run_acceptance, but this is not surfaced in tool descriptions. No tool annotations (readOnlyHint/destructiveHint) present.
Check a sentence about one observation against its evidence packet. Returns GROUNDED or REFUSED with a violation code per problem. This is the same checker that decides whether this project's own generated notes ship.
The kill gates and their verdicts, each read from the receipt rather than typed here, because a tally in a description is a number that can go stale.
Every measured field for one observation, and the reviewer note that shipped for it, including whether a generated draft was refused and why.
How many observations the queue ranks in total, named apart from the per-call cap and the review budget. Ask this rather than queue_top when the total is the whole question: the three numbers are different and two of them are 50.
The top of the physics-conditioned review queue, in rank order, with the reason code that flagged each observation.
The scalar summary of one receipt under artifacts/, with the size of each collection it holds. Never the whole file.
Output schemas not documented. LLMs cannot plan downstream calls or extract fields without knowing what queue_top, observation, check_claim, and receipt return. Baseline: 100% of A+ tools document return types.
Parameter descriptions lack constraint details. obs_id is described only as 'The SatNOGS observation ID' with no format, range, or example. limit in queue_top has min/max in schema but description does not explain the 1-50 constraint or why. Baseline: 100% of A+ tool params have descriptions with constraints.
No error handling guidance. Tools do not document what happens on invalid obs_id, missing receipt_name, or failed gate checks. LLMs have no recovery path. Baseline: pattern:recovery-guide requires 'what to do next' in error responses.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Run this repository's standing gates (scripts/gate.py) and report the verdict per check plus the tally. The one tool here that computes rather than reads, so it takes minutes, and the one that is not read-only with respect to the working tree: the gate builds the console. Nothing under artifacts/ is rewritten, because every generator the gate calls runs with --check.
run_acceptance is destructive (writes to console, takes minutes) but has no confirmation step, dry-run option, or destructiveHint annotation. Agents may invoke it unintentionally. Baseline: pattern:confirmation-request and tool annotations required for irreversible ops.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Six tools are read-only; run_acceptance is destructive. Annotations enable agents to reason about safety and retry logic. Baseline: current spec (2026-07-28) rewards tool annotations.