Loading ToolBench
All 5 tools have basic descriptions and parameter schemas, but quality is inconsistent. Descriptions are generic (avg ~60 chars, well below the 194-char baseline for A+ tools) and lack actionable context. Parameters have types but many lack descriptions entirely. No output schemas documented. Error handling guidance is absent. Tool names follow verb_noun convention but descriptions don't explain WHEN to use each tool or what distinguishes similar tools (e.g., getFlightByIdent vs getFlightDetails). No evidence of input validation, pagination support, or recovery guidance.
Retrieves information about a specific aircraft by its tail number
Searches for airports in a specific country/region
Retrieves information about a specific flight by its identifier
Retrieves detailed information about a specific flight
Retrieves flights arriving at or departing from a specific airport
Descriptions lack actionable context. 'Retrieves information about a specific flight by its identifier' (54 chars) does not explain when to use getFlightByIdent vs getFlightDetails, what fields are returned, or what prerequisites exist. Baseline for A+ tools is 194 chars with WHAT/WHEN/WHY guidance.
No output schemas documented. LLMs cannot plan downstream calls or extract required fields without knowing what each tool returns. Baseline: 100% of A+ tools have documented return types.
Parameter descriptions missing or incomplete. 'airport_code' has description 'Airport ICAO code (e.g., 'KJFK')' but no constraint on format/length. 'type' parameter in getFlightsForAirport lacks description entirely. Baseline: 100% of A+ tool params have descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | <=2025-11-25 | v2 |
No error handling guidance. No indication of retryable vs fatal errors, no recovery suggestions (e.g., 'If flight not found, try search_flights()'), no actionable error messages. Baseline: A+ tools include recovery guidance.
Tool similarity not disambiguated. getFlightByIdent and getFlightDetails both accept 'ident' and return flight data. Descriptions do not explain the difference (basic vs detailed fields, latency, cost). LLMs will guess wrong.