A multi-agent chatbot system for querying Databricks Lakehouse with web search and RAG capabilities, built with LangGraph and LangChain
AgenticLakehouse exposes three tools with significant definition gaps. The tool schemas are partially visible but lack depth in parameter documentation. Two tools (tavily_search, retrieve_docs) have minimal but present descriptions; one (databricks_mcp_tools) is delegated to an external server with no direct schema visibility in this codebase. Parameter descriptions are sparse or absent. No output schemas are documented in the visible code. Error handling patterns are not evident. The server relies heavily on LangChain adapters and external tool providers, limiting direct quality assessment.
Tools loaded from Databricks MCP server for Spark SQL query execution and Lakehouse operations
Search and return information from the vector store
Performs web searches and returns results based on user queries
databricks_mcp_tools lacks a meaningful description. Tools delegated from external MCP servers should still have LLM-optimized descriptions explaining their purpose, prerequisites, and typical usage.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required data without knowing what fields to expect.
tavily_search description ('Performs web searches and returns results based on user queries') is generic and under 60 chars. Does not explain when to use it vs retrieve_docs, how results are ranked, or what fields are returned.
retrieve_docs description ('Search and return information from the vector store') is vague. Does not explain: What data is indexed? When should this be called instead of tavily_search (local vs web)? How many results? Pagination?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 36 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 25 | - | v1 |
tavily_search and retrieve_docs both accept a 'query' parameter with only trivial description ('The search query string'). No guidance on format, length, or how queries differ between web search and RAG retrieval.
No error handling patterns visible. If tavily_search times out or retrieve_docs returns no results, the LLM receives no guidance on what to do next (retry, switch tools, ask user).
Tool naming uses underscores and is somewhat clear, but 'databricks_mcp_tools' is vague, does it create tables, run queries, manage clusters, or all of the above? This violates the principle that tool names should convey actions.
No pagination documented for tavily_search or retrieve_docs. If results are large, context window exhaustion is likely. Need limit, offset, or next_cursor in tool signatures.