Loading ToolBench
MCP server for querying Recife open datasets via HTTP/SSE and stdio transports, with SQL generation and execution capabilities powered by DuckDB and OpenRouter LLM
The server has 6 tools with reasonable structure and descriptions. All tools have descriptions (10-200 chars range), and most have input schemas. However, there are significant issues: (1) Tool descriptions in the endpoint responses contain inconsistencies with the rubric-provided summaries (e.g., list_tables has a 'required' boolean parameter that shouldn't exist); (2) Several parameters lack descriptions or have incomplete schemas; (3) Output schemas are not documented anywhere in the provided code; (4) Error handling guidance is absent; (5) No tool annotations (readonly/destructive hints). The tools follow verb-noun naming and are appropriately scoped, but the implementation falls short of production-grade quality.
Generate read-only SQL for a natural language question. Use this after list_tables/describe_table/search_schema to gather schema info. Returns SQL already validated as read-only and limited.
Get detailed column information for a specific table including column names, types, and nullability. Use this when you need to know what columns are in a table, need to know column data types, or before writing queries to ensure correct column names.
Execute a pre-written SQL query directly. Use this ONLY when you already have a complete SQL (e.g., from create_sql). The query is validated as read-only and auto-limited before execution.
List all database schemas available. Use this to see what schemas exist in the database. Most tables are in the 'public' schema. Returns a list of schema names.
List all available tables in the database with their schemas. Use this when the user asks what tables exist, what data is available, or wants to know the database structure, and before generating SQL to verify table names. Returns a list of tables with their full quoted names for use in queries.
list_tables endpoint declares spurious 'required' boolean parameter not mentioned in rubric definition. The parameter schema is malformed and adds unnecessary cognitive load for LLMs.
describe_table and list_databases also declare extraneous 'required' parameters in the endpoint response schemas that contradict the rubric-provided input definitions. These appear to be validation artifacts that should not be exposed to clients.
No output schemas documented for any tool. Rubric requires documentation of return types so LLMs know what fields to expect for downstream planning. Currently impossible to determine if execute_sql returns row_count, rows array, schema info, or other fields from the tool definitions alone.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 57 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 40 | - | v1 |
Search for tables or columns matching a keyword. Use this when looking for where specific data might be stored, searching for columns by name or concept, or finding tables related to a topic.
No error recovery guidance in any tool description. When execute_sql fails (e.g., SQL syntax error, permission denied), the LLM is not told what to do next. Descriptions should say 'If you receive an error, check column names with describe_table() first.' or similar.
create_sql tool description lacks crucial context about when to use it vs execute_sql. The dependency chain (list_tables → describe_table → create_sql → execute_sql) is implicit, not explicit. LLMs need clear ordering hints.
create_sql 'schema_context' parameter is optional but role is vague. Should explain: 'Provide output from describe_table() or search_schema() to guide SQL generation' to help LLMs understand when to call it.
search_schema 'search_term' parameter has no length, pattern, or format constraints documented. Should specify: 'Case-insensitive keyword, 1-100 characters' to prevent LLM from passing empty or massive strings.
execute_sql description mentions 'auto-limited' but limit value is not stated. LLMs cannot reason about result size without knowing the limit. Should say 'Results limited to 1000 rows' (or whatever max_result_rows is set to).
No tool annotations (readOnlyHint, destructiveHint) present. All 6 tools are read-only according to Risk field, but this is not declared in the tool definitions accessible to MCP clients. These hints help agents reason about side effects.