Loading ToolBench
Deterministic RAG output validation. Intercepts LLM extraction results, validates coverage/schema/bounds against retrieved source context via a non-LLM critic, and blocks delivery of ungrounded or vacuous results before they reach a user or downstream system.
Single tool with well-crafted description optimized for agent decision-making. Schema is present and properly typed with Zod validation. However, output schema is undocumented, LLMs cannot predict the structure of synthesis.status, keyFacts, citedChunkIds, or error fields. Parameter descriptions are adequate but lack format/constraint details. Tool name 'run_rag_reliability_pipeline' is verbose and generic ('run' is discouraged); 'validate_rag_extraction' or 'verify_rag_grounding' would be clearer. Error handling returns JSON strings rather than structured responses, forcing LLM parsing.
Deterministic RAG validation and critic orchestration. Intercepts unstructured LLM extraction output derived from external URLs, ingests source content with content-addressed chunking, and validates the extraction against retrieved context via a non-LLM critic (coverage_ratio, schema_compliance, output_bounds, determinism_self_test). Triggers when: RAG hallucination, unsupported claims, vacuous/empty extraction, or ungrounded output must be blocked before an answer is presented as fact. Output field "synthesis.status" is authoritative: "ok" = passed all critic checks, safe to present. "critic_rejected" = failed grounding validation, MUST NOT be presented as fact. No automatic correction is performed on rejection — this tool detects and blocks, it does not regenerate or fix output. Not applicable to tasks without external-source retrieval and verification.
Output schema is not documented. LLMs cannot predict the structure of the response (synthesis.status, keyFacts, citedChunkIds, error fields, coverage_ratio, schema_compliance, output_bounds, determinism_self_test results). This forces the LLM to guess at field names and types, increasing hallucination risk.
Tool name starts with generic verb 'run' rather than action-specific verb. 'run_rag_reliability_pipeline' is vague; 'verify_rag_grounding' or 'validate_rag_extraction' would immediately signal intent. Generic names reduce LLM's ability to select tools correctly.
Error responses return JSON strings (via JSON.stringify) rather than structured MCP content blocks. In mcp-server.ts, errors are wrapped in {content: [{type: 'text', text: JSON.stringify(...)}], isError: true}, forcing LLMs to parse JSON from text. Should return structured error objects with code, message, and recovery guidance.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Parameter 'mode' lacks constraint documentation. Description says 'Defaults to "structured"' but schema shows .optional() without explicit default. LLMs may not understand the fallback behavior. Should state: 'Defaults to "structured". Use "narrative" for prose output.'
No guidance on when NOT to use this tool. Description states 'Not applicable to tasks without external-source retrieval and verification' but does not explain how the LLM should detect this condition. Should add: 'Do not call if the query does not reference external sources or if the answer is already grounded in the conversation history.'