Loading ToolBench
An MCP server that lets an LLM answer questions about Malaysian elections by writing DuckDB SQL against the public ElectionData.MY data lake
Server has well-structured tool definitions with clear naming (verb_noun pattern), complete input schemas with types and descriptions, and documented output structures. However, descriptions are inconsistent in depth, some tools lack context on when/why to use them. Error handling is present but minimal; no recovery guidance or actionable error messages. Tool composition is sound (single responsibility), but parameter descriptions could be more prescriptive about constraints and formats. Output schemas are documented in code but not formally exposed in tool definitions.
Return column names and types for a lake dataset.
Execute validated read-only SQL against the ElectionData.MY lake.
List available lake datasets and whether each is streamed over HTTP.
Return a small sample of rows from a dataset.
Validate read-only SQL against lake table and voter-roll rules.
Tool descriptions lack context on when/why to use each tool. 'Execute validated read-only SQL' does not explain prerequisites (must call validate_sql first?) or when to choose execute_query vs sample_dataset. LLMs need explicit dependency hints.
Error handling is minimal. ToolError exceptions are raised but provide no recovery guidance. E.g., 'Unknown dataset: {dataset}' should suggest 'Call list_datasets() to see available options.' Agents cannot self-correct without actionable error messages.
Parameter 'sql' in validate_sql and execute_query lacks format/constraint documentation. Description should state 'Read-only SELECT queries only; no INSERT/UPDATE/DELETE/DROP allowed' and reference voter-roll restrictions explicitly.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Output schemas are documented in code (_serialize_rows, describe_dataset return structure) but not formally exposed in tool definitions. LLMs cannot see what fields to expect without reading implementation. Add explicit output schema documentation to tool docstrings.
No pagination or result-limiting guidance in tool descriptions. execute_query caps at 1000 rows but does not explain what happens if result exceeds limit or how to paginate. sample_dataset defaults to 5 rows but LLM may not understand why.