Loading ToolBench
Open-source enterprise data-agent engine with MCP server integration for database querying and management
mnemiq exposes 3 tools with explicit registration in src/mnemiq/mcp/server.py. All three have descriptions (good baseline compliance), but descriptions vary in quality. Two tools (db_read, db_write) have detailed, context-rich descriptions that guide LLM decision-making; get_schema is minimal. Input schemas are visible for all three tools with proper type declarations. However, parameter descriptions are inconsistent: db_read's 'mode' param has a helpful description and enum-like options documented in text, but the actual JSON Schema does not declare an enum constraint. db_write's single parameter 'sql' lacks type context around what SQL is permitted (INSERT/UPDATE/DELETE only). Output schemas are partially documented in prose but not formally returned in tool definitions. Error handling guidance is absent from tool descriptions, LLMs would not know what to do if a query 'defers' or if a write is refused. Security is well-designed (governance plane, identity injection, read-only declarations) but not surfaced in tool descriptions for agent awareness. Naming is clear and verb-driven (db_read, db_write, get_schema), following conventions. Composition is sound: three separate tools with single responsibilities. But descriptions do not explain WHEN to use db_read vs db_write, making tool selection harder than it needs to be. Missing: actionable recovery guidance for errors, formal schema constraints (enums), and output schema documentation.
Answer a natural-language question over the database (read-only). Returns the answer, the SQL run, and an auditable trace; defers honestly when it cannot answer. mode: 'instant' (cheapest, no retries), 'thinking' (default, self-repairing), or 'deep' (5 candidates + judge + agreement gate -- highest precision, ~6x cost).
Execute a single INSERT/UPDATE/DELETE (read/write governed separately from db_read). Refused by default: a write runs only when the identity has a write grant AND the deployment enabled writes. DDL and multi-statement input are never executed. Returns {approved, target, rows_affected, refusal, sql}; a governance plane records the result.
List the tables you are allowed to query.
Parameter constraints not formalized. db_read's 'mode' parameter documents valid values ('instant', 'thinking', 'deep') in prose but does not declare an enum constraint in the schema. LLMs will respect enums; they may hallucinate values when only prose constraints exist.
db_write parameter lacks scope/constraint documentation. The 'sql' parameter description states 'SQL INSERT/UPDATE/DELETE' but does not explain validation, rejection triggers (DDL, multi-statement), or what refusal reasons the agent should expect. LLM cannot reason about which SQL is safe to suggest.
Error handling and recovery guidance missing. db_read returns 'deferred', 'failed', 'reason_code' fields but tool description does not explain when/why these occur or what the LLM should do next (e.g., request a grant, rephrase, page operator). db_write mentions 'refusal' but provides no recovery path.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
get_schema description is minimal ('List the tables you are allowed to query'). No guidance on when to call it, what fields it returns, or what the LLM should do with the result. At 34 characters, it falls below the 50 - 200 char ideal for LLM-optimized descriptions.
Output schemas not formally documented in tool definitions. db_read returns complex nested structures (trace, lineage, narrowed, preview, verified) but the MCP tool definition does not include a formal 'outputSchema' JSON Schema. Agents must infer structure from prose or runtime discovery.
Tool descriptions do not explain mutual exclusivity or ordering. No guidance on 'call get_schema first to discover available tables, then call db_read'. LLMs will discover this through trial and error, wasting calls.