Loading ToolBench
A FastAPI-based MCP server for policy search and retrieval using semantic embeddings with Qdrant vector database, integrated with a RAG service for policy-grounded question answering
This server has critical definition gaps. Tool descriptions exist but are generic and lack LLM-optimization guidance. Input schemas are present but minimal, most parameters lack type constraints, enums, or validation hints. No output schemas are documented. Error handling is absent; tools return raw HTTP exceptions without recovery guidance. The 'ask' tool is particularly problematic: it calls external services but provides no timeout, retry, or fallback documentation. Security concerns: SQL tool accepts arbitrary dicts without parameter validation; no audit trail visible in tool definitions themselves.
Answer policy questions using RAG (Retrieval-Augmented Generation). Fetches relevant policy chunks from the MCP server, builds a prompt with system instructions, generates an answer using Ollama LLM, and extracts citations.
Health check endpoint that returns server status
Execute approved SQL stored procedures with parameter validation. Only allows whitelisted procedures (sp_GetCaseSummary). Returns procedure name, parameters, and status.
Health check endpoint for RAG service that returns server status
Readiness check for RAG service that verifies LLM temperature is 0.0, MCP server is ready, and Ollama model is warmed
Readiness check endpoint that verifies embedding model is loaded and the vector collection exists
query_sql accepts arbitrary parameter dict without type validation or schema. LLM can pass malicious payloads; no constraint documentation prevents SQL injection or invalid procedure calls.
No output schemas documented for any tool. LLMs cannot infer response structure, forcing them to guess field names and types. Breaks downstream tool chaining.
ask tool calls external MCP server with 15s timeout but provides no retry logic, fallback behavior, or error recovery guidance in description. Network failures return empty list silently.
Parameter descriptions are generic or missing. 'params' in query_sql is 'Dictionary of parameters', does not explain what keys are valid, what types they accept, or constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
Search for policy chunks using semantic similarity. Takes a query string, encodes it using a sentence transformer model, and returns matching policy chunks from the Qdrant vector database with text, source, and section metadata.
Health/ready tools are utility endpoints, not domain tools. They clutter the tool list and distract from policy search and RAG. Should be internal liveness checks, not exposed to agents.