Loading ToolBench
MCP server for analyzing and classifying insurance claims with LLM-as-judge validation
Three tools with French descriptions and basic schemas. All tools have descriptions (10-200 chars, acceptable range), but schemas use Zod with z.record(z.string(), z.unknown()) for complex objects, this is too permissive and provides zero type safety to the LLM. Parameter descriptions exist but are minimal (15-40 chars). No output schemas documented. No error handling guidance. No tool annotations (readOnlyHint, destructiveHint). Tools are READ_ONLY but this is not declared in the schema. Naming is verb-noun (analyze_, classify_, generate_) which is good, but parameter names like 'claimData' and 'classification' are vague object references without field-level documentation.
Extrait les entités structurées d'un constat de sinistre automobile : véhicules, parties, dommages et circonstances. Retourne un objet ClaimData structuré.
Classifie la sévérité d'un sinistre automobile en AUTO_PROCESS, REVIEW_NEEDED ou URGENT_ESCALATION. Utilise un LLM-as-judge (Sonnet valide Haiku) pour garantir la fiabilité.
Génère une notification courtier formelle suite à la classification d'un sinistre automobile. Inclut les détails du sinistre, la décision de traitement et les actions requises.
Schemas use z.record(z.string(), z.unknown()) for complex object parameters (claimData, classification). This provides zero type information to the LLM, it cannot infer what fields are required, their types, or their meaning. LLMs will guess at field names and types, causing runtime failures.
No output schemas documented. Tools return JSON strings but LLMs cannot see the structure of ClaimData, ClassificationResult, or the broker notification format. This forces LLMs to parse unstructured text and guess at field names for downstream tool calls.
No error handling guidance. Tools may fail (JSON parse errors in judge, LLM API failures, invalid claim data) but return no recovery hints. LLMs cannot distinguish retryable errors from user-fixable ones.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | <=2025-11-25 | v2 |
Parameter descriptions are minimal (15-40 chars). 'Objet ClaimData retourné par analyze_claim_report' tells the LLM the parameter is an object but not what fields it contains, which are required, or what they mean. Descriptions should be 50-200 chars with field-level detail.
No tool annotations. All three tools are READ_ONLY (no side effects) but this is not declared via readOnlyHint in the schema. Agents cannot distinguish safe tools from destructive ones, risking incorrect retry logic or permission checks.