Loading ToolBench
A council of language models that scores a claim against a rubric, from evidence you supply.
Quorum has well-defined tools with clear purposes and mostly complete schemas. All 5 tools have descriptions (10 - 200 chars, within baseline 34 - 392 range) and input parameters with types. However, output schemas are undocumented, tools return JSON strings without declaring the structure LLMs should expect. Parameter descriptions are present but terse (e.g., 'override the council membership for this call' lacks guidance on format or valid model names). Error handling is minimal: score_claim returns JSON errors but lacks recovery guidance. No tool annotations (readOnlyHint, destructiveHint) despite clear risk profiles (score_claim is WRITE, others READ_ONLY). Naming is strong (verb_noun pattern: score_, describe_, verify_, list_, health). Composition is good, each tool has one responsibility. Missing: documented output schemas, parameter constraints (enums for models), and actionable error messages.
Show the rubric the council will use, exactly as the members will see it.
Check that the server has what it needs before anyone depends on it.
List saved council runs, newest last.
Score one claim with a council of models, against a rubric, using only the evidence given.
Recompute the hashes on a saved run and report whether it was altered.
Output schemas undocumented. All tools return JSON strings without declaring field structure, types, or required fields. LLMs cannot plan downstream calls or extract data reliably.
Parameter 'models' lacks enum or format constraint. Description says 'override the council membership' but does not specify valid model names, format (e.g., 'provider/model-name'), or how to discover available models.
Error handling lacks recovery guidance. score_claim returns JSON errors (e.g., 'evidence is required') but does not tell the LLM what to do next or how to self-correct.
No tool annotations. score_claim is destructive (WRITE, saves transcripts) and should declare destructiveHint=true. All READ_ONLY tools should declare readOnlyHint=true for agent optimization.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Parameter descriptions are terse. 'optional note on how the claim is being used' (context) and 'inline rubric YAML' (rubric_yaml) lack format guidance, examples of valid input, or constraints.