Streamlit application that integrates with dbt's MCP server for Semantic Layer queries, using Snowflake's LLM (Cortex) to infer semantic layer parameters from natural language prompts and execute compiled SQL queries.
This server exposes 4 tools (list_metrics, get_dimensions, get_entities, query_metrics) for dbt Semantic Layer integration via HTTP. The code is visible and tool definitions can be extracted from streamlit_v1_single_sl.py. However, critical gaps emerge: (1) Tool descriptions are present but minimal (10-30 chars, well below the 50-200 char LLM-optimized baseline). (2) Parameter descriptions are sparse or absent for most inputs, e.g., 'where' parameter has no explanation of SQL syntax expectations, 'order_by' lacks format guidance. (3) No output schemas are documented, the code extracts results via extract_text() and json.loads(), but the expected response structure is never formalized for the LLM. (4) Error handling is present but generic ('Could not read content', 'Non-JSON data'), no recovery guidance (e.g., 'Try calling list_metrics first'). (5) All tools are READ_ONLY, which is good, but this constraint is not formally annotated in tool definitions (no toolAnnotations/readOnlyHint per the spec). The tool names follow verb_noun convention correctly, but parameter design lacks constraints and validation guidance.
Get dimensions available for a given set of metrics
Get entities available for a given set of metrics
List all available metrics from the dbt Semantic Layer catalog
Query metrics with specified dimensions, filters, and grouping
Output schemas completely undocumented. No tool description explains what fields or structure the LLM should expect in responses. Code shows json.loads() parsing but no schema specification visible to the LLM.
Descriptions are too brief (10-105 chars vs. 50-200 char baseline). They lack WHEN to call each tool, WHAT it returns, and HOW it differs from similar tools. LLMs cannot reliably select the right tool when descriptions are this sparse.
Parameter constraints missing. 'where' parameter accepts a SQL-like string but the description does not explain syntax (WHERE id = 1? WHERE metric > 100?). 'order_by' expects 'objects' but format is unclear. 'limit' has no bounds documented (e.g., 1-10000). LLMs will guess and fail.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2026-07-28+ | v2 |
No error recovery guidance. Code shows generic error messages ('Could not read content from list_metrics tool call', 'Non-JSON data'). These tell the LLM nothing about what went wrong or what to try next. Per pattern:recovery-guide, errors should guide next steps.
Tool annotations absent. All tools are READ_ONLY (no side effects), but this is not formally declared via readOnlyHint or tool metadata. Per current spec (2026-07-28), tool annotations improve agent reasoning about idempotency and safety.
Parameter descriptions lack type/format examples. 'metrics' is described as 'List of metric names' but the LLM does not know the exact format (string array? lowercase? spaces allowed?). 'order_by' says 'list of objects' but no example structure given.