Loading ToolBench
A tool-using AI agent powered by local Qwen 3.5 (via Ollama) that queries and manages RisingWave through MCP tool calls
This server demonstrates a pattern-based tool design with basic definitions, but has critical quality gaps across naming, parameter documentation, and error handling. All 6 tools are explicitly defined in agent.py with JSON Schema structures (properties and required arrays visible). Naming is verb_noun compliant (list_, run_, describe_, table_, execute_, insert_) and self-documenting. Descriptions are present for all tools and most parameters, ranging 64-159 chars for tool descriptions (within the 34-392 baseline). However, there are significant weaknesses: (1) Parameter descriptions are minimal or generic, e.g., 'Name of the table' lacks format/constraint details. (2) No output schemas are documented, callers do not know what fields to expect from JSON responses. (3) Error handling is present but minimal, errors return JSON with 'status' and 'error' fields but provide no recovery guidance or classification (retryable vs. fatal). (4) Security concerns: the `describe_table` and `table_row_count` tools use bare string interpolation in SQL queries (f"WHERE table_name = '{table_name}'"), which is vulnerable to SQL injection if the LLM passes a malicious table name. (5) The `run_sql_query` tool performs only a regex check ('startswith("SELECT")') to enforce read-only access, trivial bypasses exist (e.g., 'SELECT * FROM t; DROP TABLE x;'). No actual result limits or pagination are implemented despite the baseline requirement for large result sets.
Show the column names and data types of a specific table.
Execute a DDL statement (CREATE TABLE, DROP TABLE, etc.) against RisingWave.
Execute an INSERT statement to add data to a RisingWave table.
List all user tables in the RisingWave public schema. Use this when the user asks to see their tables.
Execute a read-only SQL SELECT query against RisingWave and return the results. Use this to answer questions about data in tables.
Get the number of rows in a specific table.
No output schemas documented. Tools return JSON with 'results', 'status', 'error', 'tables', 'columns', etc., but the response structure is never formally declared. LLMs cannot predict which fields will be present or their types, forcing unreliable downstream tool chaining.
SQL injection vulnerability in describe_table and table_row_count. Table names are interpolated directly into SQL queries using f-strings (e.g., f"WHERE table_name = '{table_name}'") without escaping. An LLM could be tricked into passing a malicious table name like "'; DROP TABLE users; --" causing data loss.
Weak access control on run_sql_query. The tool checks only if the query starts with 'SELECT' using a simple string comparison. Trivial bypasses: 'SELECT * FROM t; DROP TABLE x;' or 'SELECT * UNION ALL DROP TABLE t;' would pass the check and execute destructive commands.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 48 | - | v1 |
Parameter descriptions lack format and constraint details. E.g., 'The SQL SELECT query to execute' does not specify that only SELECT is allowed or what happens if a DML/DDL statement is passed. 'Name of the table' lacks format constraints or length limits.
Error handling provides no recovery guidance. Errors return JSON with 'status': 'error' and an 'error' string, but do not classify the error (retryable? user-fixable? fatal?), do not suggest next steps, and do not include invalid values or constraint violations. Per pattern, errors should guide the LLM on what to do next.
No pagination or result limits. The run_sql_query tool executes arbitrary SELECT queries and returns all rows. If a table has 100k rows, all 100k are returned, bloating the response and exhausting context windows. Baseline requires pagination and per-result limits (20-50 items).
No destructive operation safeguards. execute_ddl and insert_data perform state-altering operations (CREATE, DROP, INSERT) with no confirmation, dry-run, or undo mechanism. Per pattern, irreversible operations should support a confirmation step or dry-run to prevent catastrophic agent errors.
No tool annotations. The tools lack readOnlyHint, destructiveHint, or idempotentHint fields as specified in the current MCP spec. These annotations help clients reason about side effects and retry safety.