LangGraph-based data analysis agent with multi-source support (PostgreSQL, SQLite, RAG knowledge base), intent classification, tag-based context matching, and interactive graph execution
ChatBL exhibits severe definition quality gaps across all dimensions. Tool definitions are inferred from prompts.py rather than explicit MCP registration code. Descriptions are present but generic (avg ~80 chars, well below 194-char baseline). Input schemas lack proper JSON Schema structure with type definitions for most parameters. Critical tools like run_python_script_tool and analysis_worker accept free-form strings with no validation constraints, enums, or format guidance. No output schemas documented. Error handling is absent, no recovery guidance, categorization, or actionable error messages. Security concerns: run_python_script_tool executes arbitrary Python code with no sandboxing, rate limiting, or permission gates visible. Tool composition is poor: analysis_worker and db_worker are catch-all tools that duplicate functionality of more specific tools (sales_predict_tool, reviews_stars_correlation_test_tool), forcing LLMs to reason about overlapping responsibilities.
Terminate the conversation and return final response to user
Perform data analysis, visualization, prediction, and knowledge base retrieval
Ask user for clarification when intent is ambiguous
Query business database (PostgreSQL) for data analysis
Get current datetime for handling relative time expressions like 'this month', 'last quarter'
List available tables and their structure in the business database
Analyze correlation between product reviews and star ratings
Execute Python scripts for data analysis, visualization, and predictions
run_python_script_tool accepts arbitrary Python code with no sandboxing, validation, or permission gates. Poses critical security risk and violates pattern:secret-injection and pattern:permission-gate.
Tool definitions inferred from prompts.py rather than explicit MCP registration. No visible tool registration code with proper schema declarations.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. Violates pattern:tool and pattern:response-shaper.
Overlapping tool responsibilities: analysis_worker, db_worker, and sqlite_worker duplicate functionality of specific tools (sales_predict_tool, reviews_stars_correlation_test_tool). LLMs waste reasoning cycles deciding between them.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 35 | <=2025-11-25 | v2 |
Predict future sales based on historical data
Search and retrieve documents from the knowledge base (RAG)
List available tables and their structure in the uploaded SQLite database
Query uploaded SQLite tables for data analysis
Translate data analysis requirements into Python plotting scripts
Validate data quality and business logic in database and uploaded tables
Free-form string parameters (query, script, requirement) lack enums, format constraints, or validation guidance. LLMs hallucinate invalid values. No error recovery guidance provided.