Loading ToolBench
Compiles and executes YAML semantic models as analytical SQL with REST API and Arrow Flight SQL support
The OrionBelt Semantic Layer server exhibits severe quality issues across naming, descriptions, and schema consistency. While the domain (semantic layer query compilation) is well-understood, the tool definitions show evidence of duplication (10 tools appear 2-3 times with identical or near-identical names and schemas), incomplete parameter descriptions, and missing input/output schema documentation. No tool annotations (readOnlyHint, idempotentHint) are present despite all tools being READ_ONLY. Descriptions are present but vary wildly in quality, some are under 20 characters, others conflate similar tools. Parameters lack formal constraints (enums, ranges), forcing LLMs to guess valid values. No error handling guidance, no recovery patterns, and no structured output schema documentation.
Compile an advanced query with filters, ordering, and HAVING clauses.
Compile a semantic query to SQL. Dimensions and measures must be exact business names from the model.
Get the full semantic model structure: data objects, dimensions, measures, metrics. Call this first to understand what is available before compiling queries.
Explain the lineage of a dimension, measure, or metric. Shows which data objects, columns, joins, and expressions contribute to the named artefact.
Get the join graph showing how data objects (tables) are connected. Returns nodes (data objects) and edges (joins) with cardinality and join columns.
List all supported SQL dialects with their capabilities. Supported: bigquery, clickhouse, databricks, dremio, duckdb, mysql, postgres, snowflake.
List all dimensions in the semantic model. Dimensions are categorical or temporal attributes used for grouping and filtering (e.g. Country Name, Sales Date, Product Category).
CRITICAL: Massive tool duplication. 30 tools advertised, but only ~10 unique operations. Tools 1-10, 11-20, 21-30 represent the same semantic functions repeated verbatim with trivial naming variations (PascalCase → snake_case → repeat). This confuses agent planning, wastes context tokens, and violates the single-responsibility principle. Example: 'Describe Model' (Tool 1), 'describe_model' (Tool 11), 'describe_model' (Tool 21) are identical.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | D | 53 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 59 | - | v1 |
List all measures in the semantic model. Measures are numeric aggregations computed from data object columns (e.g. Total Sales, Sales Count, Avg Unit Price).
List all metrics in the semantic model. Metrics are derived calculations built from measures (e.g. Profit Margin, YoY Growth). Types: derived, cumulative, period_over_period.
Search for dimensions, measures, or metrics by name or synonym.
Compile a semantic query to SQL. Dimensions and measures must be exact business names from the model.
Compile a semantic query to SQL. Dimensions and measures must be referenced by their exact business names as returned by describe_model, list_dimensions, and list_measures.
Compile an advanced query with filters, ordering, and HAVING clauses. Use this for queries that need WHERE filters, HAVING filters, ORDER BY, or other advanced features not covered by compile_query.
Compile an advanced query with filters, ordering, and HAVING clauses.
Get the full semantic model structure: data objects, dimensions, measures, metrics. Call this first to understand what is available before compiling queries.
Get the full semantic model structure: data objects, dimensions, measures, metrics. Call this first to understand what is available before compiling queries.
Explain the lineage of a dimension, measure, or metric. Shows which data objects, columns, joins, and expressions contribute to the named artefact.
Explain the lineage of a dimension, measure, or metric. Shows which data objects, columns, joins, and expressions contribute to the named artefact.
Get the join graph showing how data objects (tables) are connected. Returns nodes (data objects) and edges (joins) with cardinality and join columns.
Get the join graph showing how data objects (tables) are connected. Returns nodes (data objects) and edges (joins) with cardinality and join columns.
List all supported SQL dialects with their capabilities. Supported: bigquery, clickhouse, databricks, dremio, duckdb, mysql, postgres, snowflake.
List all supported SQL dialects with their capabilities. Supported: bigquery, clickhouse, databricks, dremio, duckdb, mysql, postgres, snowflake.
List all dimensions in the semantic model. Dimensions are categorical or temporal attributes used for grouping and filtering (e.g. Country Name, Sales Date, Product Category).
List all dimensions in the semantic model. Dimensions are categorical or temporal attributes used for grouping and filtering (e.g. Country Name, Sales Date, Product Category).
List all measures in the semantic model. Measures are numeric aggregations computed from data object columns (e.g. Total Sales, Sales Count, Avg Unit Price).
List all measures in the semantic model. Measures are numeric aggregations computed from data object columns (e.g. Total Sales, Sales Count, Avg Unit Price).
List all metrics in the semantic model. Metrics are derived calculations built from measures (e.g. Profit Margin, YoY Growth). Types: derived, cumulative, period_over_period.
List all metrics in the semantic model. Metrics are derived calculations built from measures (e.g. Profit Margin, YoY Growth). Types: derived, cumulative, period_over_period.
Search for dimensions, measures, or metrics by name or synonym. Use this when the user mentions a concept and you need to find the matching artefact (e.g. "sales" might match a measure with synonym "sales").
Search for dimensions, measures, or metrics by name or synonym.
CRITICAL: NO input schemas visible. All 30 tools report 'Input: {}' or have schemas only in the text description, not as formal JSON Schema. This prevents clients from validating agent calls and leaves LLMs guessing at parameter types and constraints.
HIGH: No tool annotations (readOnlyHint, idempotentHint, destructiveHint). ALL 30 tools are READ_ONLY and idempotent (no side effects), but the server provides zero annotations. Per the 2026-07-28 spec, tool annotations enable agents to reason about safety. Without them, agents cannot distinguish safe-to-retry tools from irreversible ones, increasing error recovery latency.
HIGH: Inconsistent parameter descriptions and missing constraints. 'Compile Query' (Tools 6, 16, 26) each have slightly different parameter descriptions. None declare enums for the 'dialect' parameter despite a fixed set (bigquery, clickhouse, databricks, dremio, duckdb, mysql, postgres, snowflake). LLMs must hallucinate valid dialects instead of being guided by constraints. Example: Tool 6 says 'default: postgres' but Tools 16 and 26 have identical descriptions with no example.
HIGH: Missing output schema documentation. No tool documents what it returns. E.g. 'Describe Model' says 'Get the full semantic model structure' but provides no schema showing whether the response includes arrays of dimensions, measures, metrics, or nested objects. LLMs cannot plan downstream tool calls or extract necessary IDs without knowing the output shape.
MEDIUM: Parameter descriptions under 20 chars are vague. Tool 9 'Search Model' has parameter 'query' with description 'Search term (case-insensitive).', under 20 chars and lacks guidance on what the LLM should search for (dimension names only? measure synonyms? metric types?).
MEDIUM: No error handling or recovery guidance. No tool describes what happens on failure (invalid dimension name, unsupported dialect, malformed JSON query). LLMs receive raw errors with no actionable next steps. Per pattern 'recovery-guide', errors must tell agents what to do next.
MEDIUM: Ambiguous tool naming ('Compile Query' vs 'Compile Advanced Query'). The distinction, basic query vs one with filters/HAVING, is unclear from names alone. 'compile_query' should have a variant name like 'compile_query_with_filters' or 'compile_filtered_query' to make the distinction obvious. Agents may select the wrong tool.
MEDIUM: Inconsistent parameter types across variants. Tool 6 'Compile Query' has dimensions/measures as comma-separated strings; Tool 16 has them as arrays. Tool 26 is array-based with improved description. This inconsistency forces agents to remember which variant uses which format, increasing errors.