Loading ToolBench
An MCP server for BigQuery exploration with cost guardrails and a built-in NL-to-SQL eval harness.
Server has 7 well-named tools with consistent verb_noun patterns (list_*, describe_*, search_*, estimate_*, run_*). All tools have descriptions (avg 95 chars, within 10-1024 baseline). Input schemas are present and typed for all tools. However, parameter descriptions are minimal (1-2 sentences), output schemas are not documented, and error handling lacks recovery guidance. No tool annotations (readOnlyHint, destructiveHint). Missing pagination for list_datasets and list_tables despite baseline expectations. Overall solid foundation but lacks depth in LLM-optimization patterns.
Schema and metadata for a single table.
Dry-run a query; return bytes_scanned and estimated_usd.
List all datasets in the configured BigQuery project.
List all tables in a given dataset.
Execute a read-only query. Refuses non-SELECT statements and queries whose dry-run estimate is over the byte cap; returns rows or a structured error.
Return up to n sample rows from a table.
Output schemas not documented. LLMs cannot plan downstream calls or extract fields without knowing return structure. E.g., list_datasets returns list[dict[str, object]] but field names (dataset_id, created_time, etc.) are undocumented.
No pagination support on list_datasets and list_tables. Baseline expects page/offset + limit + total_count for list tools. Large result sets will blow context window.
Error handling lacks recovery guidance. BigQueryError.to_dict() returns {error, message, details} but does not tell LLM what to do next (retry? ask user? fatal?). No error classification (retryable vs user-fixable vs fatal).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Fuzzy-match a term against all column names; return top hits.
No tool annotations. run_query and estimate_cost should declare readOnlyHint=true. This helps clients optimize caching and retry logic.
Parameter descriptions are minimal (1-2 sentences). Baseline expects 50-200 chars with format/range/constraint details. E.g., 'n' in sample_table lacks guidance on valid range (1-1000? 1-10000?).