Loading ToolBench
RAG-powered data catalog, lineage, and change-impact assistant with REST and MCP interfaces
13 tools with basic schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. All tools are READ_ONLY (good for safety), but descriptions are generic and lack LLM-optimized guidance on when/why to use each tool. Parameter descriptions exist but are minimal (avg ~40 chars, below 72-char baseline). No error handling guidance, no output schemas documented, no tool annotations. Tools like generate_query and validate_query have reasonable descriptions, but most metadata/lineage tools (search_by_table, get_lineage, compare_data_assets) lack specificity on what fields are returned and how to chain them. This is typical community-server quality.
Analyze where a data asset is used and its impact score (metadata).
Assess the blast radius of a proposed change.
Compare two assets and surface shared upstream/downstream lineage (metadata).
Explain what a SQL statement does in plain English (rule-based).
Generate a PostgreSQL SELECT from a natural-language description (LLM-backed).
Get detailed metadata plus upstream/downstream for a single asset (metadata).
Get upstream/downstream lineage for an asset (metadata).
Output schemas not documented. Tools like get_lineage, analyze_data_usage, and compare_data_assets return complex nested objects (upstream/downstream arrays, impact_score, etc.) but the MCP tool definitions do not declare output schema. LLMs cannot plan downstream calls or extract fields without knowing the response structure.
Parameter descriptions are minimal and lack constraint guidance. 'top_k' in search_data_assets and search_similar_queries has no description of valid range (e.g., 1-100). 'direction' in get_lineage accepts 'upstream|downstream|both' but this enum is not declared in schema or description. LLMs will hallucinate invalid values.
No error handling guidance. Tools return empty objects or null on failure (e.g., analyze_data_usage returns empty arrays if impact_analyzer is None) with no indication of what went wrong or how to recover. LLMs cannot distinguish between 'no results' and 'service unavailable'.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | <=2025-11-25 | v2 |
List data assets owned by a person or team (metadata).
Look up catalog metadata and lineage for a table (metadata).
Semantic search over the catalog/lineage store (embeddings).
Find similar saved queries/reports by description (embeddings).
Suggest optimizations for a SQL query (rule-based).
Validate SQL syntax and safety using rule-based checks (no LLM).
Tool descriptions lack LLM-optimized guidance on when to use each tool. 'search_data_assets' vs 'search_by_table' vs 'search_by_owner' are semantically similar; descriptions do not explain the distinction or when to prefer one over another. This forces LLMs to guess and wastes reasoning cycles.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). While all tools are READ_ONLY, this is not declared in the MCP schema. Agents cannot infer safety properties and may apply unnecessary caution or retry logic.