Loading ToolBench

Reading the current corpus…

retrieval-lens MCP server · D · ToolBench

retrieval-lens

MCP server that audits RAG retrieval — logs what chunks the model saw, scores, queries, and sources before any answer was generated.

2025-06-18+STDIO4 tools

53/100

Definition Quality70

Protocol Readiness42

Supportability31

Spec posture

Inferred effective spec 2025-06-18+·current baseline 2026-07-28

Static source inference · medium confidence · detected: Logging

Deprecated protocol patterns detected

Summary

retrieval-lens defines 4 tools with reasonable structure but exhibits significant gaps in output schema documentation and parameter descriptions. Tool names follow verb_noun convention (retrieval_observe, retrieval_query, retrieval_diff, retrieval_stats) which is clear, but descriptions lack LLM optimization and several parameters are under-documented. Input schemas are present for all tools with proper type definitions, but output schemas are entirely absent from the tool definitions, critical for agent planning. Error handling and recovery guidance is not visible in the codebase. The LlamaIndex adapter example shows practical integration but doesn't improve tool definition quality.

Tools (4)

retrieval_diffread onlysource verified68/100

Compare two retrieval runs side by side to find missing chunks, shared chunks, and score movement between runs.

retrieval_observewritesource verified75/100

Capture a RAG retrieval run, including query, chunks, scores, sources, and ranks, for later audit and diff workflows.

retrieval_queryread onlysource verified70/100

Replay stored retrieval runs so agents can inspect exactly which chunks, scores, sources, and ranks reached the model.

retrieval_statsread onlysource verified68/100

Aggregate retrieval quality and volume metrics across stored runs, including score distributions, top sources, and daily trends.

Issues (5)

CRITICALtool ↗

Output schemas completely undocumented. No tool defines what it returns, field names, or types. LLMs cannot plan downstream actions or validate responses.

retrieval_observeretrieval_queryretrieval_diffretrieval_stats

HIGHrecovery-guide ↗

Error handling and recovery guidance absent. No indication how tools fail, what errors mean, or what the agent should do next (retry? try alternative? ask user?).

retrieval_observeretrieval_queryretrieval_diffretrieval_stats

MEDIUMconstrained-input ↗

retrieval_query 'limit' parameter lacks min/max bounds. Description says '0-100, default 10' in natural language but not enforced in schema. Schema should have minimum: 0, maximum: 100.

retrieval_query

MEDIUMtool-description ↗

retrieval_diff 'match_by' enum is documented but schema shows enum values without explicit description of what 'source' vs 'content_hash' matching means in practice.

Recommendations

Document output schemas for all 4 tools. Define the exact fields, types, and structure returned by each. Example: retrieval_observe should document 'returns {success: bool, run_id: string, inserted: bool}'; retrieval_query should document 'returns {runs: [{run_id, query, pipeline_tag, created_at, chunks: [{content, score, source, rank}]}], total_count: number}'.
Add error handling guidance to each tool description. State common failure modes and recovery steps, e.g. 'retrieval_query: If no runs found matching filters, returns empty array. Try broader pipeline_tag or remove time filters to expand results.'
Enforce schema constraints in JSON Schema, not just descriptions. Add 'minimum': 0, 'maximum': 100 to retrieval_query.limit; add explicit descriptions to enum values in retrieval_diff.match_by explaining the difference between source and content_hash matching strategies.
Expand retrieval_observe description to explicitly state it WRITES records to the audit database and persists them for later querying. Clarify that duplicate run_ids are rejected (idempotent).
Add pagination details to retrieval_query and retrieval_stats output docs. If results are truncated, include a 'next_cursor' or 'has_more' field so agents know when to paginate.
Document what fields are returned in chunk records (content, score, source, rank) and clarify the rank meaning (1-indexed position in retrieval order).

Spec posture evidence

Inferred effective spec: 2025-06-18+.

Relies on Logging (deprecated) - log to stderr or use OpenTelemetry

Score history

Overall score trend

First recorded score · v2 rubric

53/100

Scored	Grade	Overall	Spec posture	Rubric
2026-09-23	D	53	2025-06-18+	v2