Loading ToolBench
A FastAPI-based customer service chatbot for Cymbal Air airline that uses LangGraph and LangChain to provide an agentic interface for flight booking, ticket management, and customer inquiries. Integrates with a Toolbox backend for tool execution and uses Google Vertex AI for LLM capabilities.
Three tools with basic schemas but significant quality gaps. insert_ticket and validate_ticket have identical parameter sets (6 fields, all strings) with minimal descriptions (10-15 chars each). execute_sql is a critical security risk: accepts arbitrary SQL with only a 24-char description. No output schemas documented. No error handling guidance. Parameter descriptions lack format constraints, ranges, or validation rules. Tool descriptions are functional but generic (50-80 chars), missing WHEN/WHY context. No enum constraints despite domain-specific values (airport codes, airline names). Naming is verb-first (good), but descriptions do not explain prerequisites, return structure, or failure modes.
Executes SQL queries against the database for data initialization and export operations
Inserts a flight ticket for the logged-in user after validation and confirmation
Validates flight data and returns corrected ticket information before booking confirmation
execute_sql accepts arbitrary SQL with no input validation, sanitization, or permission checks. Description is 24 chars ('Executes SQL queries...'), far below the 50-200 char baseline for LLM-optimized descriptions. No guidance on allowed operations, error recovery, or security constraints.
Parameter descriptions are 10-15 characters (e.g., 'Departure airport code'). Rubric baseline is 72 chars. No format constraints (airport codes are 3-letter IATA codes), no enums, no validation rules. LLMs cannot infer valid values.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. Rubric baseline: 100% of A+ tools have documented return types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | 2026-07-28+ | v2 |
insert_ticket and validate_ticket have identical parameter sets but no documented difference in behavior or output. Tool descriptions do not explain when to call validate_ticket vs insert_ticket, or what validate_ticket returns.
No error handling guidance. No recovery hints (e.g., 'If validation fails, check airport codes'). No categorization of errors as retryable, user-fixable, or fatal. Rubric requires: 'Error responses must tell the LLM what to do next.'