Loading ToolBench
Tools for querying a synthetic e-commerce sales dataset (customers, products, orders). Use list_tables/describe_table to discover the schema, aggregate/time_series for common group-by and trend questions, and run_sql (read-only, automatically row-capped) for anything else.
Server has 5 tools with complete schemas and descriptions. Naming follows verb_noun convention (list_tables, describe_table, run_sql, aggregate, time_series). Descriptions are present and actionable (avg ~120 chars). All parameters have type definitions and descriptions. However, descriptions lack LLM-optimized guidance on WHEN to use each tool vs alternatives, and error handling lacks recovery guidance. Output schemas are documented in code but not formally exposed in tool definitions. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only operations.
Group-by aggregation on one table, e.g. order count by region, or average unit_price by category. agg is one of: sum, avg, count, min, max. metric is required unless agg is "count".
Describe a table's columns and row count.
List every table in the dataset.
Run a read-only SELECT/CTE query. Automatically row-capped; writes and DDL are rejected.
Bucket a date column into a day/week/month time series, aggregating value_column (or counting rows if omitted).
No tool annotations despite all tools being read-only. Missing readOnlyHint metadata that would help LLMs understand safety properties and avoid unnecessary confirmation steps.
Error handling lacks recovery guidance. ToolError exceptions are raised but descriptions do not explain what the LLM should do next (e.g., 'unknown table' error should suggest calling list_tables first).
aggregate and time_series descriptions lack guidance on WHEN to use them vs run_sql. LLMs cannot distinguish between 'use aggregate for simple group-by' vs 'use run_sql for complex queries' without explicit comparison.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2025-06-18+ | v2 |
Output schemas are documented in code (tools.py return dicts) but not formally exposed in MCP tool definitions. LLMs cannot see the structure of returned 'groups' or 'points' arrays without reading implementation.
run_sql description mentions 'automatically row-capped' but does not state the actual limit. LLMs cannot reason about result size or plan pagination without knowing the cap value.