Loading ToolBench
A motor claim intake application demonstrating shared draft editing between a claimant and an AI agent through WebMCP tools, with real-time synchronization and policy-driven validation.
ClaimReady has 10 tools with mixed quality. Strengths: strong naming conventions (verb_noun pattern), detailed descriptions for most tools (avg 150+ chars), and explicit input schemas for 2 tools. Weaknesses: 8 of 10 tools lack visible input schemas in source code (only apply_claim_patch and record_supporting_details show schemas); descriptions for tools 2-5, 7, 9-10 are inferred from eval files rather than explicit registration; no output schemas documented; error handling guidance absent. The server demonstrates good intent (e.g., apply_claim_patch's baseRevision pattern prevents race conditions) but lacks the rigor expected of production tools.
Set one or more fields on the motor claim draft, all of them or none of them, as a single revision. Call read_claim_state first and send the revision it reported as baseRevision: if the person on the page has corrected something since you read, the patch is refused and nothing changes, so read again. Dates are YYYY-MM-DD, damage_zone is a clock position 1 to 12, vehicle_drivable is true or false. Filing the finished claim is a control on this page and is not exposed as a WebMCP tool.
Check whether the claim is covered under the policy schedule. Returns coverage status and any exclusions that apply.
Describe the current state of the motor claim in natural language, including all fields, their values, and who set each one.
Get roadside assistance options available for the claim. This tool is only published when the claim indicates the vehicle is not drivable.
Get an estimate of repair costs for the damage described in the claim based on the damage zone and severity.
8 of 10 tools lack visible input schemas in source code. Only apply_claim_patch and record_supporting_details show explicit JSON Schema definitions. Tools 2-5, 7, 9-10 are defined in eval files (probe_assertions.mjs, browser_probe.mjs) with no schema visible, making it impossible to verify parameter types, constraints, or required fields.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does check_coverage return coverage_status + exclusions, or a boolean?). Descriptions mention what tools return (e.g., 'coverage status and any exclusions') but lack formal schema.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
Get the list of intake requirements that must be satisfied before the claim can be filed, based on the current claim state and the insurer's policy rules.
Read the complete current state of the motor claim draft, including all field values, who set each field, the current revision number, which fields are pinned, and what is still missing.
Read any evidence notes or supporting documents that have been attached to the claim by the claimant or other parties.
Record supporting details for the claim, such as witness information or additional notes. This tool is built declaratively from HTML form attributes rather than registered programmatically.
Validate the current claim state against the policy rules and return any validation errors or warnings.
Tool definitions inferred from eval files rather than explicit registration. Tools 2-5, 7, 9-10 are defined in evals/probe_assertions.mjs and evals/browser_probe.mjs with no corresponding registration code visible in src/webmcp/. Per HARD SCORING RULES, inferred tools cap at 50 overall.
No error handling guidance. Descriptions do not explain what happens on failure (e.g., if check_coverage finds no matching policy, does it return null, an error, or a message?). LLMs cannot plan recovery without explicit error classification.
record_supporting_details schema incomplete. Input schema shows witness_name and base_revision but description says 'Record supporting details' (plural). Schema does not enumerate all supported fields or explain what 'supporting details' means beyond witness info.