Loading ToolBench
A Model Context Protocol server for ArangoDB with async support, multi-database management, graph operations, and backup/restore capabilities
The server defines 46 tools with basic descriptions and input schemas, but quality is inconsistent across the suite. Core ArangoDB operations (arango_query, arango_insert, arango_update) have reasonable schemas with typed parameters and descriptions, meeting baseline documentation standards. However, the tool portfolio exhibits significant design debt: (1) Naming conventions are inconsistent, some tools use clear verb-noun patterns (arango_insert, arango_remove) while others are vague (arango_search_tools, arango_switch_workflow). (2) Descriptions are sparse and generic; most lack context about when to use the tool, prerequisites, or what state changes occur. For example, arango_insert says 'Insert a document into a collection' but does not explain error conditions, return format beyond 'return_new' flag, or whether this is idempotent. (3) Output schemas are not documented, the rubric requires structured response schemas so LLMs know what fields to expect; tool definitions here only show input schemas. (4) Approximately 40% of tools (arango_search_tools, arango_list_tools_by_category, arango_get_tool_usage_stats, arango_unload_tools, arango_switch_workflow, arango_advance_workflow_stage, arango_get_active_workflow, arango_list_workflows) appear to be meta-operations about the tool system itself rather than ArangoDB operations, which violates single-responsibility and composition principles. (5) No error handling guidance is present, tools do not describe retryable vs. fatal conditions, invalid input handling, or recovery paths. (6) Parameter descriptions exist but lack constraints, ranges, or format specifications. For instance, batch_size accepts an integer with no min/max bounds; collection accepts a string with no indication of valid formats or naming rules. The server is functional but exhibits mediocre production readiness.
Add an edge to a graph.
Add an edge definition to a graph.
Add a vertex to a graph (alias for arango_insert).
Add a vertex collection to a graph.
Advance the workflow to the next stage.
Backup collections to JSON files.
Backup a graph to a directory.
Backup multiple named graphs to a directory.
Output schemas not documented. Tool definitions specify input schemas but never document what fields are returned, forcing LLMs to guess response structure and increasing hallucination risk.
Meta-operation tools violate single responsibility. Tools like arango_search_tools, arango_list_tools_by_category, arango_get_tool_usage_stats, and arango_unload_tools operate on the tool system itself, not on ArangoDB. These should be removed or moved to a separate introspection layer.
Workflow state management tools (arango_switch_workflow, arango_get_active_workflow, arango_list_workflows, arango_advance_workflow_stage) lack context about what workflows exist, what stages are valid, and how state transitions work. Descriptions are generic and provide no transition rules or error conditions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 46 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 44 | - | v1 |
Insert multiple documents into a collection.
Update multiple documents in a collection.
Create a new collection in the database.
Create a new graph in the database.
Create an index on a collection.
Create a schema for document validation.
Get the status of the current database.
Delete an index from a collection.
Explain an AQL query execution plan.
Get the currently active workflow.
Get database resolution status and information.
Get the currently focused database.
Get usage statistics for tools.
Calculate statistics about a graph.
Traverse a graph starting from a vertex (alias for arango_traverse).
Insert a document into a collection.
Insert a document with reference validation.
List all available configured databases.
List all non-system collections in the database.
List all graphs in the database.
List all indexes for a collection.
List all available tools organized by category.
List all available workflows.
Execute an AQL query with optional bind vars and return rows.
Build and execute a query with a builder interface.
Profile an AQL query for performance analysis.
Remove a document from a collection.
Restore a graph from a backup directory.
Search for available tools by name or description.
Set the focused database for multi-database operations.
Find the shortest path between two vertices in a graph.
Switch the active workflow context.
Traverse a graph starting from a vertex.
Unload tools from memory.
Update a document in a collection.
Validate a document against a schema.
Validate the structural integrity of a graph.
Validate reference integrity between collections.
Parameter constraints missing. Numeric parameters like batch_size, doc_limit, and depth lack min/max bounds. String parameters like collection, graph, and field lack format specifications or naming rules. This invites invalid input from LLMs.
Error handling guidance absent. No tool describes retryable vs. fatal conditions, invalid input handling, or recovery paths. LLMs cannot self-correct without guidance on what went wrong and why.
Destructive operations lack confirmation or dry-run support. Tools like arango_remove and arango_delete_index perform irreversible actions with no confirm_before_execute pattern or preview step to prevent agent mistakes.
Duplicate/alias tools reduce clarity. arango_graph_traversal and arango_add_vertex duplicate arango_traverse and arango_insert respectively. These aliases confuse LLM tool selection and waste reasoning cycles; consolidate to single canonical names.
Description quality is generic and brief. Most descriptions are 30-60 characters and lack context about when to use the tool, what it modifies, prerequisites, or common error conditions. Baseline for A-grade tools is 50-200 characters with clear intent.