Loading ToolBench
A RAG (Retrieval-Augmented Generation) CLI tool for managing embedding, reranking, and generation servers with document indexing capabilities
rag-cli exposes 20 tools with significant definition gaps. Most tools lack proper descriptions (many under 20 chars or generic). Input schemas are present but sparse, many parameters lack type information or descriptions. Tool names are action-oriented (start, stop, delete, index) but descriptions fail to explain WHEN to use each tool or what distinguishes similar operations. No output schemas documented. Error handling is absent, tools provide no recovery guidance. Security concerns: database connection objects passed as parameters (tools 17-20) expose internal state. Composition issues: tools 4-6 (start/stop/restart) are near-duplicates; tools 14-16 (index workflows) lack clear differentiation.
Check if a named server is healthy
Delete chunks from database by collection and/or document
Delete indexed chunks for a collection or specific document
Check if a document is completely indexed in the database
Ensure a specific set of servers are running and stop others
Ensure a server or operation class is ready (start if needed, handle exclusivity)
Ensure database schema exists with vector and sparse vector support
Database connection objects exposed as parameters (tools 17-20: ensure_schema, doc_is_complete, delete_chunks, store_chunks). Passing 'conn' objects violates secret-injection and tool-gateway patterns, internal state leaks into logs and traces. Should accept connection string or use server-side injection.
Descriptions missing or under 20 characters for 8 tools (start, stop, restart, start_all, stop_all, ensure_schema, doc_is_complete). LLMs cannot determine when to select these tools or how they differ from similar operations. Baseline: 194 chars average.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. Tools return unstructured responses (e.g., status() returns dict with 'running', 'pid', 'healthy' but no schema). Baseline: 100% of A+ tools document return types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | 2026-07-28+ | v2 |
Find the URL of a running server by name
Health check endpoint for SPLADE server
Index a collection of markdown documents or a single document into the vector database
Index chunks from a JSON file into the vector database with embeddings
Restart a named server preset
Generate sparse embeddings using SPLADE model for input texts
Start a named server preset
Start all default server presets
Start an arbitrary embedding or rerank server with custom model path and port
Get status of all managed servers (running, port, PID, health)
Stop a named server preset
Stop all running server presets
Store chunks with dense and sparse embeddings into the database
Destructive tools (delete_workflow, delete_chunks) lack confirmation or dry-run support. Agents can irreversibly delete indexed data without safeguards. No error guidance for recovery.
Duplicate/overlapping tool responsibilities: start/stop/restart (tools 4-6) are near-identical; index_collection_workflow and index_json_workflow (tools 14-15) both index but with different input formats. No clear guidance on when to use each. Violates single-responsibility principle.