Loading ToolBench
Pre-flight query cost & result-size guardrails for AI agents across BigQuery, Snowflake, and Databricks
Four tools with complete input schemas, descriptions, and type definitions. Naming follows verb_noun pattern (check_, describe_, estimate_, run_). Descriptions are detailed and actionable (150-300 chars), explaining WHAT, WHEN, and WHY. Parameters are well-typed with clear descriptions. Output schemas are documented via Pydantic types. Error handling via @as_tool_error decorator. Main gaps: no enum constraints on Engine/warehouse_size/edition parameters (invites hallucination), no explicit pagination guidance, and descriptions could be more concise for LLM token efficiency. Tool annotations present (read_only_hint, open_world_hint) but destructiveHint missing on run_query_bounded despite WRITE risk.
Verify credentials/connectivity for an engine without running any real query or dry-run estimate. Call this once after configuring a new engine (or when a real tool call fails) to get a fast, clear yes/no signal instead of debugging via trial queries.
Declare which cost signals are exact vs. approximate for the given warehouse engine. Call this before estimate_query_cost or run_query_bounded to understand how much to trust the accuracy_tier on their responses for this engine.
Estimate the cost of a SQL query before running it. ALWAYS call this before running an expensive-looking query. The response's accuracy_tier tells you how much to trust the number: PRECISE (exact), UPPER_BOUND (real cap, may overstate), HEURISTIC (rough). warehouse_size (Snowflake/Databricks only, e.g. "SMALL", "X-Large") and edition (Snowflake only, e.g. "enterprise") default to the smallest/standard tier if omitted — set them to match the warehouse you actually run this query on, or the dollar figure will understate cost on a larger warehouse.
Run a query only if its pre-flight cost estimate is within your given bounds; refuses otherwise (check result.status — "refused" means it did NOT run and result.hint explains why). NOTE: unlike estimate_query_cost, a successful call here has a real monetary/quota side effect — don't call this repeatedly without inspecting the result of each call. warehouse_size/edition (see estimate_query_cost) default to the smallest/standard tier if omitted - set them to match the warehouse you actually run on, since max_estimated_cost_usd is checked against the estimate they produce.
Engine parameter lacks enum constraint. Accepts 'bigquery|snowflake|databricks' but no formal enum in schema, LLMs may hallucinate invalid values like 'redshift' or 'postgres'.
warehouse_size and edition parameters accept free-form strings ('SMALL', 'X-Large', 'enterprise') with no enum or validation. Descriptions mention examples but no formal constraint, LLMs will guess invalid sizes like 'MEDIUM' or 'pro'.
run_query_bounded marked as WRITE risk but lacks destructiveHint=True in ToolAnnotations. Tool description warns of 'real monetary/quota side effect' but annotation does not reflect this, agents may not recognize the irreversible nature.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | <=2025-11-25 | v2 |
Descriptions are 150-300 chars, above the 10-200 char sweet spot for LLM token efficiency. Verbose explanations of accuracy_tier and warehouse defaults waste tokens; could be condensed and moved to parameter descriptions.
No pagination or result-limiting guidance. If estimate_query_cost or run_query_bounded return large result sets (e.g. detailed cost breakdowns), no limit or cursor mechanism is documented.