Loading ToolBench
Three well-defined tools with clear, specific descriptions (150 - 200 chars each) and complete input schemas. All parameters are typed and described. Tool names follow verb_noun convention (validate_*, cusip_to_isin). Output schemas are implicit but reasonable for validation/conversion tasks. No tool annotations (readOnlyHint, idempotentHint). Offline, keyless design is a strength for reliability.
Convert a CUSIP to its ISIN by prefixing the country code (default "US") and appending the computed ISIN check digit. E.g. CUSIP "037833100" -> "US0378331005".
Validate a CUSIP (9-char US/Canada securities identifier, e.g. "037833100"). Checks structure + the check digit.
Validate an ISIN (International Securities Identification Number, ISO 6166 — 12 chars, e.g. "US0378331005"). Checks structure + the mod-10 check digit and returns the ISO country prefix and NSIN. Keyless/offline; does NOT resolve the issuer.
Output schemas not documented. LLMs cannot plan downstream calls or extract fields without knowing what validate_isin returns (e.g., does it return country_prefix, nsin, check_digit?). Implicit schemas force LLMs to guess.
Descriptions include example values ('US0378331005', '037833100'). Replace with formal constraints (regex patterns, length rules) in parameter descriptions.
No tool annotations. Tools are read-only and idempotent but lack readOnlyHint and idempotentHint metadata. Agents cannot infer safety properties without explicit annotations.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Error messages are descriptive but lack recovery guidance. E.g., 'Not a well-formed ISIN' tells the LLM the input is wrong but not what to do next. Add hints like 'Expected format: 2 letters + 9 alphanumerics + 1 check digit.'
cusip_to_isin 'country' parameter has no validation constraint in description. Should state: 'ISO 2-letter country code (e.g., US, CA). Defaults to US if omitted.'