Loading ToolBench
RAG project template with FastMCP and pgvector for loading CSV files, cleaning data, chunking text, generating embeddings, and performing semantic search over a vector database
The RAG Server has 5 tools with basic definitions but significant quality gaps. All tools have descriptions present (10 - 250 chars) and input schemas visible in the source code. However, parameter descriptions are minimal or absent, output schemas are not formally documented, and error handling lacks recovery guidance. The naming follows verb_noun convention, which is good, but parameter constraints are weak. No enum constraints, no min/max specifications, and several parameters lack descriptions or have trivial ones. Output responses are free-text strings rather than structured objects, forcing LLMs to parse unstructured data. Error messages are basic ('Error: {str(e)}') and do not guide the agent on what to do next. The server does not implement tool annotations (readOnlyHint, destructiveHint, idempotentHint), missing a critical signal for agents to reason about side effects and idempotency.
Clear all indexed documents and chunks from the database.
Get statistics about the indexed documents and chunks.
Load a CSV file, clean it, chunk it, embed it, and store it in the vector database.
Search the vector database for relevant context and return results.
Quick semantic search for documents matching a query.
No output schemas documented. All tools return free-text strings (str type) instead of structured JSON objects. This forces LLMs to parse unstructured output, wasting tokens and increasing hallucination risk. Responses should be dicts/lists with typed fields (e.g., query_rag should return {results: [{source_file, chunk_text, similarity}]} not a formatted string).
No tool annotations. Tools marked as READ_ONLY, WRITE, or DESTRUCTIVE in the metadata are not exposed via MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint). Agents cannot reason about side effects or idempotency without this signal.
Error handling lacks recovery guidance. All exception handlers return 'Error: {str(e)}' (e.g., in ingest_csv, query_rag, get_stats). These raw error messages tell agents nothing about what to do next. Errors should be categorized and include actionable recovery hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 54 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 59 | - | v1 |
Missing parameter descriptions. In query_rag, source_file parameter description is present but top_k lacks explanation of why the agent should change the default. In get_stats and clear_documents, input is empty {} with no explanation of what they do. Parameter descriptions should explain what each input controls and when to use it.
No input validation or constraints. Parameters like top_k accept unbounded integers (no min/max declared). text_column is a free-form string with no enum or pattern. file_path has no validation against path traversal attacks. Constraints must be explicit in schema and enforced in implementation.
No pagination support. query_rag and search_documents return top_k results but do not support pagination (limit/offset, cursor). If the vector database contains thousands of chunks, returning 5-10 results per call is fine, but the tools should declare max limits in descriptions and implement pagination if results exceed a threshold.
Missing confirmation for destructive operations. clear_documents() has no dry-run or confirmation step. An agent can accidentally call it and nuke the database. Irreversible operations should require a separate confirmation tool or a confirm_mode parameter.
Tool composition gap: search_documents is a thin wrapper around query_rag. Both take a query and top_k and call the same underlying logic. This violates the single-responsibility principle and wastes agent reasoning. Remove search_documents or merge it into query_rag with a parameter variant.