Loading ToolBench
Advanced Retrieval-Augmented Generation (RAG) server providing semantic search, chunk search, graph analysis, and project/document management capabilities. Agents and LLMs can use RAG/meta-tools for code understanding, impact analysis, and cross-modal queries.
The server exposes 5 tools with significant quality gaps. Tools are defined in multiple files (mcp-server.js and mcp-server.ts) with inconsistent schemas. Most critically: (1) Description quality is mixed, some tools have adequate guidance, but others are terse or lack essential context. (2) Parameter descriptions are sparse or missing entirely on several tools. (3) Output schemas are present but generic (all return {result: object}), providing no guidance on what fields to expect. (4) The unified "action" pattern across graph, project, and vector_database tools creates confusion about parameter handling and valid operation names. (5) Error handling is minimal, responses lack recovery guidance. The STDIO transport caps the overall protocol score at 50, regardless of definition quality.
Unified tool for graph operations: fetch the full graph, get neighbors of a node, or search for nodes/edges by substring or property. Use the "action" parameter to select the operation. IMPORTANT: Always pass the project name using the 'project' field (not 'name').
List all available projects. Use this to discover valid project names before using other project tools.
Unified tool for project management actions. The FIRST action any LLM should take is list_projects to discover available projects. Actions: - list_projects: List all projects (no arguments required). - get_project: Retrieve a single project's metadata. Pass the project name in the 'project' field. - add_project: Add a new project. Pass the project name in the 'name' field and path in the 'path' field. - delete_project: Delete a project. Pass the project name in the 'name' field. - list_files: List files for a project. Pass the project name in the 'project' field. - list_documents: List documents for a project. Pass the project name in the 'project' field. IMPORTANT: For all actions except add_project and delete_project, pass the project name using the 'project' field (not 'name').
Upload a new document to the project. Note: For richer project understanding and semantic search, use the RAG/meta-tools provided by this server. Always pass the project name using the 'project' field (not 'name').
Unified 'action' parameter design conflates multiple independent tools into one. The 'project' tool mixes list_projects, get_project, add_project, delete_project, list_files, list_documents, six distinct responsibilities. The 'graph' tool similarly mixes get_graph, graph_neighbors, search_nodes, search_edges. This violates the Single Responsibility Principle and forces LLMs to reason about which 'action' enum value is correct instead of directly calling a verb_noun tool (e.g., list_files, delete_project, search_nodes as separate tools).
Output schemas are uniformly vague ({type: 'object', properties: {result: {type: 'object'}}, required: ['result']}). LLMs cannot determine what fields 'result' contains, their types, or whether additional context (like pagination, IDs for chaining, or metadata) is included. Agents cannot plan downstream calls without seeing the structure of returned data.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Unified tool for all vector database operations. Actions: query (semantic search), chunk_search (search document chunks), list_chunks (list all indexed chunks), explain (explain a vector search result), reindex_document (recompute embeddings for a document). Use the action parameter to select the operation. IMPORTANT: Always pass the project name using the 'project' field (not 'name').
Parameter descriptions are often missing or extremely terse. Example: 'query' parameter in graph tool has description 'Substring/property for search.', does not clarify scope (nodes only? edges only? both?), case sensitivity, regex vs literal substring, or whether wildcards are supported. Parameter 'result' in vector_database/explain tool is described only as 'Result object for explain' without documenting its required structure.
No enumeration constraints on free-form string parameters. 'project' parameter appears in many tools but is unconstrained, LLMs can hallucinate project names. The 'action' enums (in graph, project, vector_database) are defined, but the toolSchemas in mcp-server.js show simpler tool definitions (query, chunk_search, graph_neighbors, etc.) that lack enums on 'action' parameters where they should exist.
Error handling is absent. The callRagApi() function throws McpError(ErrorCode.InternalError, error.message) for any backend failure, but provides no recovery guidance. Agents do not know if the error is retryable, user-fixable (e.g., invalid project name), or fatal. No distinction between 'project not found' (user should select a valid project) and 'backend service unreachable' (retry later).
Tool definitions in mcp-server.js differ significantly from the documented schemas in the assignment. The mcp-server.js toolSchemas array defines 15 tools (query, chunk_search, graph_neighbors, upload_document, explain, add_project, delete_project, reindex_document, update_indexer_status, clear_query_cache, clear_general_cache, list_projects, get_project, list_files, list_documents, get_graph), but the assignment documents only 5 unified tools (list_projects, graph, project, vector_database, upload_document). This discrepancy suggests either incomplete implementation or misaligned documentation.
Parameter naming inconsistency: The project tool description explicitly warns to use 'project' field instead of 'name', and similarly for graph and vector_database tools. However, the schema shows 'name' is used for add_project and delete_project. This is documented but forces LLMs to memorize which tools use which parameter names, a recipe for off-by-one errors. Standardize on a single parameter name across all operations on the same resource type.
No pagination support. Tools like list_projects, list_files, list_documents, and get_graph do not expose limit/offset or cursor parameters. The assignment notes this is a 'unified tool' for multiple actions, but if list_files can return hundreds of files, the response will blow the context window. Agents cannot page through large result sets.
Tool descriptions lack dependency hints and prerequisites. Example: project tool says 'FIRST action any LLM should take is list_projects', but this guidance is buried in the description and not enforced. There is no explicit indication in the graph or vector_database tools that they require a valid project to exist first. Agents may attempt operations on non-existent projects and hit cryptic backend errors.