Loading ToolBench
Composite MCP server for dbt model analysis and data product quality assessment with GitHub repository support
Data Product Hub server exhibits significant definition quality gaps. Of 12 tools, only 8 have visible input schemas in the provided code. Tool descriptions are present but generic (average ~80 chars, well below the 194-char production baseline). Parameters lack consistent descriptions, especially in tools that interact with GitHub APIs and AI services. Critical issues: no output schema documentation, inconsistent parameter documentation, missing error recovery guidance, and tools like 'generate_dalle_image' and 'create_github_mcp_server' that expose implementation details rather than user-facing actions. The server prioritizes internal workflows (MCP server lifecycle tools) over end-user data product operations.
Analyze a specific dbt model for quality and best practices
Analyze a specific dbt model with AI-powered suggestions (requires OpenAI API key)
Enhanced dbt model analysis with Git context from connected Git MCP server
Analyze all dbt models in a GitHub repository
Comprehensive data product quality assessment (future: will aggregate multiple tools)
Check metadata coverage across all dbt models
Get list of connected external MCP servers
Get lineage information for all models in the dbt project
No output schema documentation. Tools return responses but the provided code shows no documented return types, field names, or structure. LLMs cannot plan downstream calls or parse results reliably.
Vague parameter descriptions. 'prompt' in generate_response tools lacks detail on format, length limits, or expected content structure. 'repo_url' in validate_github_repository provides no guidance on error handling when repo is invalid.
API credentials exposed as tool parameters. get_openai_api_key and get_environment_secret accept repo_url to retrieve secrets. This design risks secrets in logs. Should use server-side injection via environment or vault.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 12 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 48 | - | v1 |
Validate a dbt project structure and configuration
Mixed concerns in tool names. 'start_mcp_server' and 'start_mcp_server_hostable' are infrastructure lifecycle tools, not data product tools. This muddies the tool namespace and confuses LLMs about the server's primary purpose.
No error recovery guidance. Descriptions do not explain what to do when operations fail (e.g., 'repo not found', 'OpenAI API unavailable'). LLMs receive no direction on retry or fallback strategies.
get_repository tool has WRITE risk but no confirmation or dry-run mode documented. Cloning repos to disk is a side effect, agents should not execute this without explicit intent confirmation.
Inconsistent naming conventions. Most tools use verb_noun (analyze_dbt_project, validate_github_repository), but start_mcp_server and create_github_mcp_server expose internal MCP server lifecycle concerns rather than data product operations that end users care about.