Loading ToolBench
ChatBI is an intelligent data conversation assistant that combines LLM with data query tools, RAG, and chart generation capabilities. It provides a FastAPI backend with tools for SQLite queries, natural language to SQL conversion, chart generation, data export, and knowledge base search.
ChatBI exposes 9 tools with moderate structural issues. Tool naming is mostly verb-first but lacks clarity in several cases (e.g., 'high_charts_json' is awkward; 'DuckDuckGoSearchRun' is a library name, not an MCP tool name convention). Descriptions exist for all tools but are inconsistent in detail and actionability, some are generic or in Chinese without English translations. Schemas are partially visible in source: parameters have types and descriptions, but output schemas are not explicitly documented anywhere in the provided code. Error handling is minimal; no recovery guidance is provided. Tool composition has issues: execute_sqlite_query accepts raw SQL, creating injection risk; no dry-run or confirmation patterns; no pagination for search results. Security concerns around SQL execution without sanitization. Overall falls in the 'fair to poor' range (C - D) due to incomplete schemas, inconsistent descriptions, and lack of error guidance.
Search the web using DuckDuckGo search engine
Search for database schema details
Execute a SQLite query on a fixed database and return the results as JSON. Use this tool to interact with the SQLite database.
Export chart as PNG image file
Export data to CSV and/or Excel formats
Export data and charts as PDF report
获取指定时区的当前时间，返回格式如 2025-09-06 15:30:00。参数 timezone 例如 'Asia/Shanghai'、'UTC'、'America/New_York'。
Tool naming convention: 'DuckDuckGoSearchRun' is a library class name, not MCP tool naming. Should be 'search_web' or 'duckduckgo_search' following verb_noun convention.
No output schemas documented. Tools return results but callers cannot predict response structure. execute_sqlite_query returns {'status', 'result'} but structure of 'result' is unclear for SELECT vs non-SELECT queries.
SQL injection vulnerability in execute_sqlite_query: raw user-provided 'query' parameter is passed directly to sqlite3.execute() with no sanitization, validation, or parameterized query support. LLM can be tricked via prompt injection to pass malicious SQL.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 34 | - | v1 |
Use LLM to generate Highcharts JSON config from a list of numbers and chart type.
Use LLM to convert natural language text to a SQLite query.
No error recovery guidance. execute_sqlite_query returns raw SQLite errors (e.g., 'syntax error') with no hint about what to try next. Does not categorize errors as retryable vs fatal.
Descriptions are inconsistent: some tools have Chinese descriptions only (e.g., get_time_by_timezone's full description is in Chinese). MCP servers should provide English descriptions for LLM portability.
Parameter 'table_schema' in text2sqlite_query is optional and defaults to empty string. Unclear when/why to pass it. No guidance on what structure it expects.
No pagination support in database_schema_rag or DuckDuckGoSearchRun. If schema is large or search returns 500+ results, response blows context window.
No idempotency or confirmation pattern for destructive tools. export_chart_png, export_data_csv_xlsx, export_report_pdf all write files. No dry-run mode or confirmation step to prevent accidental overwrites.
high_charts_json description mentions 'Use LLM to generate' but does not clearly state when to use this vs alternatives. No guidance on when Highcharts is appropriate.
database_schema_rag has a vague description ('Search for database schema details'). Does not explain what structure it returns, whether it indexes table names, columns, constraints, or all three.