Loading ToolBench
Three read-only tools with basic descriptions and schemas. Tool names follow verb_noun convention (read_query, list_tables, describe_table). Descriptions are present but generic (avg 150 chars, below the 194-char baseline for A+ tools). Input schemas are properly typed with JSON Schema. However, parameter descriptions lack actionable constraints (e.g., no regex pattern for SQL queries, no enum for fetch_all rationale). Output schemas are not documented, LLMs cannot infer the structure of returned dictionaries. Error handling is present but minimal (FileNotFoundError, ValueError) with no recovery guidance. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only. Security: SQL injection is mitigated via parameterized queries and statement validation, but the validation logic (checking for 'select'/'with' prefixes) is fragile and could be bypassed. Overall: functional but below production baseline.
Get detailed information about a table's schema. Args: table_name: Name of the table to describe Returns: List of dictionaries containing column information: - name: Column name - type: Column data type - notnull: Whether the column can contain NULL values - dflt_value: Default value for the column - pk: Whether the column is part of the primary key
List all tables in the Messages database. Returns: List of table names in the database
Execute a query on the Messages database. Args: query: SELECT SQL query to execute params: Optional list of parameters for the query fetch_all: If True, fetches all results. If False, fetches one row. row_limit: Maximum number of rows to return (default 1000) Returns: List of dictionaries containing the query results
Output schemas not documented. LLMs cannot infer structure of returned dictionaries (e.g., describe_table returns list of dicts with 'name', 'type', 'notnull', 'dflt_value', 'pk' fields, but this is only visible in docstring, not in tool schema).
Parameter descriptions lack actionable constraints. 'query' param has no regex pattern, length limit, or examples of valid SELECT syntax. 'row_limit' has no min/max bounds (default 1000 is large; no guidance on safe ranges).
Error messages lack recovery guidance. FileNotFoundError and ValueError are raised but do not tell the LLM what to do next (e.g., 'Database not found. Ensure SQLITE_DB_PATH is set correctly.').
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | <=2025-11-25 | v2 |
No tool annotations despite all tools being read-only. Missing readOnlyHint and idempotentHint annotations would help agents understand safety and retry semantics.
SQL injection validation is fragile. Checking query_lower.startswith(('select', 'with')) can be bypassed (e.g., ' SELECT' with leading spaces, or comments). Use a proper SQL parser or stricter regex.