Loading ToolBench
Trust protocol for AI agents. Prove capability through calibration, not claims.
caliber has 7 well-named tools with clear, domain-specific descriptions (avg 156 chars). All tools have input schemas with typed parameters and descriptions. However, output schemas are underdocumented, caliber_card, caliber_trajectory, and caliber_integrity return complex dicts without field-level documentation. Error handling is minimal; tools return generic strings or error dicts without recovery guidance. No tool annotations (readOnlyHint/destructiveHint). Parameter descriptions are adequate but lack format constraints (e.g., confidence range 50-99 or 0.50-0.99 is stated but not enforced via schema). Composition is sound, tools chain logically (predict → verify → card/summary/list). Security: no secrets exposed as params; agent/prediction_id are user-provided strings, not tokens.
Generate a Trust Card from accumulated predictions. Returns the Trust Card as structured JSON with confidence buckets, domain stats, danger zones, strength zones, and significance tests.
Check the prediction record for gaming signatures. Calibration alone can be farmed with easy predictions. This runs deterministic statistics — Brier score decomposition (reliability / resolution / uncertainty), confidence and domain concentration, duplicate claims, predict->verify latency, batch-import share — and returns advisory flags with evidence. There is deliberately no aggregate score.
List predictions.
Record a prediction before verifying it.
Quick calibration stats without a full Trust Card.
Show how calibration changes over time. Returns trajectory with snapshots at regular intervals and trends (improving/declining/stable).
Output schemas underdocumented. caliber_card, caliber_trajectory, and caliber_integrity return complex dicts (Trust Card, trajectory snapshots, integrity report) with no field-level schema or type documentation. LLMs cannot reliably extract nested fields or plan downstream operations.
No tool annotations. All tools lack readOnlyHint/destructiveHint/idempotentHint. caliber_predict and caliber_verify are WRITE operations; caliber_card, caliber_summary, caliber_list, caliber_trajectory, caliber_integrity are READ-ONLY. Annotations enable agents to reason about side effects and retry safety.
Minimal error handling. Tools return generic strings ('Prediction recorded: ...') or bare error dicts ('error': 'No verified predictions yet.'). No recovery guidance, no actionable next steps, no error classification (retryable vs user-fixable vs fatal).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Record the outcome of a prediction.
Parameter constraints not enforced in schema. confidence is described as '50-99 or 0.50-0.99' but schema has no minimum/maximum. interval defaults to 10 with no bounds. LLMs may pass invalid values; tools must validate and return clear constraint violations.
caliber_list returns minimal fields (id, claim, confidence, domain, outcome) but omits timestamps, agent name, and notes. Agents cannot reconstruct full prediction context or sort by recency without a follow-up call.