Loading ToolBench
Read-only access to a business database. Call list_tables first, then describe_table to learn the columns, then run_sql for anything else. Every query is capped and audited; writes are impossible.
Five well-named, verb-prefixed tools with clear descriptions (avg 156 chars, within 10-1024 baseline). All tools have documented input schemas with typed parameters and descriptions. Output is JSON-structured. However, parameter descriptions lack format/constraint details (e.g., 'limit' has no min/max bounds), and error handling is minimal, responses return JSON errors but lack recovery guidance. No tool annotations (readOnlyHint, etc.). Missing output schema documentation in tool docstrings. Composition is sound: each tool does one thing, and outputs chain well (table names from list_tables feed into describe_table and search). Security is strong: read-only enforcement, SQL injection prevention via parameterization, and audit logging mentioned in run_sql description.
Show the columns, types and masked fields of one table, plus 3 sample rows.
List the tables the model is allowed to read, with their row counts.
Run one read-only SELECT and return the rows. Only a single SELECT (or WITH ... SELECT) statement is accepted, results are capped, PII columns are masked, and every call is written to the audit log.
Case-insensitive substring search in one column - no SQL needed.
Profile one column: nulls, distinct values, min/max, top 5 values. Useful as a first look at unfamiliar data before writing any SQL.
Parameter constraints not documented in descriptions. 'limit' parameter in search() has no min/max bounds stated; 'sql' in run_sql() lacks format guidance (e.g., 'single SELECT or WITH...SELECT only'). LLMs cannot infer constraints from code, they rely on description text.
Error responses lack recovery guidance. run_sql() returns JSON with 'error' and 'hint' fields, but describe_table() and search() return bare error messages without suggesting next steps (e.g., 'unknown column, try describe_table() first').
Output schemas not documented in tool docstrings. Callers must infer response structure from code (e.g., run_sql returns {sql, columns, rows, row_count, truncated, elapsed_ms}). Explicit schema documentation in docstrings enables LLM planning.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | <=2025-11-25 | v2 |
No tool annotations (readOnlyHint, idempotentHint). All five tools are read-only and idempotent, but this is not declared in the tool metadata. Agents cannot infer safety properties without explicit hints.
Result capping mentioned in run_sql description but not in search() or summarize_column(). Agents need to know limits are enforced to avoid planning for unbounded results. Baseline: cap at 20-50 items and state in description.