Loading ToolBench
Answers natural-language questions about the ShopSphere e-commerce database by generating SQL with a fine-tuned specialist model, executing it read-only, and returning the actual rows -- not just the query text.
Two tools with basic descriptions but significant schema and parameter documentation gaps. nl_to_sql has a lengthy description (>200 chars) but lacks formal input schema with type/description for the 'question' parameter. sql_specialist_stats has minimal description (47 chars) and empty input schema. Neither tool documents output structure or error recovery paths. No parameter constraints (enums, ranges) visible. Error handling returns raw error strings without recovery guidance.
Answer a natural-language question about the ShopSphere e-commerce database. Generates a SQL query, executes it read-only, and returns the result rows (capped at 200) along with the SQL used, so the answer is grounded in the actual data rather than the model's guess. The database covers customers, employees, products, orders, order_items, reviews, and support_tickets -- see schema/schema.sql for full column definitions. Only read queries are permitted; any write/DDL attempt is rejected before execution.
Return observability stats for the currently active predictor: call count, average latency, success rate, and total cost so far.
Input schemas lack parameter descriptions. nl_to_sql 'question' parameter has no description in schema; sql_specialist_stats has empty input object. LLMs cannot infer parameter meaning from names alone.
Output schemas not documented. nl_to_sql returns dict with 'sql', 'row_count', 'rows', 'truncated', 'error' fields but no formal schema. sql_specialist_stats returns dict with unknown structure. LLMs cannot plan downstream calls without knowing response shape.
Error responses lack recovery guidance. Errors return raw strings ('predictor error: ...', 'rejected unsafe SQL: ...') without actionable next steps. Pattern requires 'Try X instead' or error classification (retryable/user-fixable/fatal).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | <=2025-11-25 | v2 |
sql_specialist_stats description is too brief (47 chars, below 50-char minimum for clarity). Does not explain when to call it, what 'stats' means, or what fields are returned. Baseline: A+ tools average 194 chars.
No input validation or constraint documentation. nl_to_sql accepts any string; no max length, format, or safety constraints documented. No enum/pattern constraints visible in schema.