Loading ToolBench
Streaming LLM chat over governed data: typed tools, MCP, RBAC, and a Kafka governance spine.
Ledger MCP provides 9 well-named, analytically-focused tools with clear descriptions and structured schemas. Tool names follow verb_noun convention (list_columns, describe_column, aggregate, timeseries). Descriptions are detailed (100-250 chars) and explain WHEN to use each tool. Input schemas are comprehensive with type definitions, enums, and constraints (e.g., grain enum in timeseries, maxLength on filters). However, output schemas are not documented, LLMs cannot predict response structure for downstream chaining. Error handling is absent from visible code; no guidance on retryability or recovery. Security posture is unclear (no visible permission gates or audit logging). Tools are well-composed for analytical workflows but lack the response documentation needed for production agent integration.
Compute one or more metrics, optionally grouped and filtered. The general-purpose analysis tool: use it for totals, averages, and breakdowns.
Compare two explicit time windows, with per-day rates beside the totals. Use this for every before-and-after question rather than running two queries and subtracting. Windows of unequal length cannot be compared on totals; this returns both lengths and the per-day rates so the comparison holds even when they differ.
Count rows matching a set of filters. If nothing matches, the result explains which filter eliminated everything and what values are actually present.
Get the full profile of one column. Returns null fraction, cardinality, range, quantiles, sample values, and any caveat about how the column is recorded.
Bin a numeric column into a histogram. The upper tail is clipped by default, because a single extreme value would otherwise put every row in one bin.
List the columns available in the dataset, with their types and descriptions. Call this first when unsure what the data contains, or to check a column name before using it.
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination) for downstream tool chaining or result extraction. Agents cannot plan multi-step workflows without guessing response format.
No error handling guidance visible. Tools lack recovery hints (e.g., 'If column not found, call list_columns() first'). Agents cannot self-correct on failures.
plot tool accepts result_id but no documentation of how result caching works or what result_id format is valid. Agents cannot reliably construct valid IDs.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Render a cached result as a chart. Plot renders a cached result, never a fresh query. If it re-queried, a non-deterministic tie-break or a shifted boundary could make the chart disagree with the prose above it.
Compute metrics bucketed over time, optionally split into a few series. Use this for trends, seasonality, and any before-and-after comparison.
Rank the values of one column by a metric and return the best or worst n. Prefer this for "busiest", "highest", or "which X had the most Y". When ranking by an average, set min_group_rows so a single outlier cannot win.
Pagination not visible in tool definitions. Large result sets (e.g., list_columns on wide datasets) could exceed context windows. No limit or offset parameters documented.
No permission gates or scope declarations visible. Tools are READ_ONLY but no explicit permission model (e.g., 'read:data', 'read:catalog') documented for RBAC enforcement.