Eval-driven SQL reliability for AI agents. Safe SQL tool layer that validates and executes read-only queries with policy enforcement.
QueryPilot exposes 4 database-focused tools with basic descriptions and minimal parameter documentation. All tools have names starting with action verbs (ask_, search_, validate_, execute_) and descriptions present, but descriptions are brief (10-50 chars) and lack context on when to use each tool vs. alternatives. Input schemas are visible but sparse: each tool accepts only 1-2 string parameters with minimal type information. No output schemas are documented in the source. Error handling is not evident from the code sample. The tools appear well-intentioned for SQL safety but lack the depth of documentation and schema rigor expected in production-grade agent tools.
Ask a database question through QueryPilot's safe SQL flow.
Validate and execute read-only SQL safely.
Search tables and columns relevant to a question.
Validate SQL and return policy metadata.
Descriptions are too brief (10-50 chars). Rubric baseline is 194 chars avg; descriptions lack context on when to use each tool, prerequisites, and what they return.
Output schemas are not documented. LLMs cannot plan downstream calls or extract data without knowing what fields to expect from each tool.
Parameter descriptions are missing or minimal. The 'question' param in ask_database and 'query' param in search_schema lack guidance on format, length, or expected structure.
No error handling guidance visible. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next on failure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Tool composition unclear. The relationship between search_schema, validate_sql, and execute_sql is not documented. LLMs cannot determine the intended workflow or when to call each.