Loading ToolBench

Reading the current corpus…

PyRag MCP server · D · ToolBench

← Browse

PyRag

github.com/nateislas/PyRag

Python Documentation RAG System for AI Coding Assistants

<=2025-11-25STDIO2 tools

55/100

Definition Quality74

Protocol Readiness42

Supportability33

Spec posture

Inferred effective spec <=2025-11-25·current baseline 2026-07-28

Static source inference · medium confidence · detected: Logging

Deprecated protocol patterns detected

Summary

PyRAG exposes 2 tools with reasonable descriptions and partially specified schemas. Tool names follow verb_noun conventions (search_docs, diagnose). Both tools have descriptions exceeding the 20-character minimum. However, schema completeness is mixed: search_docs has a detailed multi-parameter schema with types and descriptions; diagnose is simpler but complete. Parameter descriptions are present and actionable. Output schemas are not documented, critical gap for LLM planning. Error handling is minimal; no recovery guidance provided. Security is acceptable (read-only tools with no secret injection risks). The server is functionally competent but lacks the output schema documentation and error recovery patterns expected of production-grade tools.

Tools (2)

diagnoseread onlysource verified71/100

Comprehensive diagnostic tool for PyRAG server health and connectivity. This tool runs various diagnostic checks to help troubleshoot issues: - health: Basic server health (imports, Python version, environment) - database: ChromaDB connectivity and data availability - server: Complete server status including PyRAG initialization - all: Run all diagnostic checks (default)

search_docsread onlysource verified77/100

Unified Python documentation search with flexible response formats. This is the main RAG tool that searches through indexed Python library documentation with support for different response formats and optional streaming for complex queries.

Issues (5)

CRITICALtool ↗

Output schema not documented for either tool. LLMs cannot plan downstream tool calls or extract returned fields without seeing the response structure.

search_docsdiagnose

HIGHrecovery-guide ↗

No error handling guidance. Tools return errors without explaining what the LLM should do next (retry, ask user, adjust parameters, etc.). This violates recovery-guide pattern.

search_docsdiagnose

HIGHconstrained-input ↗

search_docs accepts 'library' as optional but provides no enum of valid libraries. LLM may hallucinate invalid library names. Should either constrain to known libraries or document lookup behavior.

search_docs

MEDIUMpaginated-result ↗

No pagination support documented for search_docs despite potentially returning 20+ results. Large result sets risk context window exhaustion and degrade LLM reasoning.

search_docs

Recommendations

Add explicit output schema documentation for search_docs (e.g. returns {results: [{source, snippet, relevance_score}], total_count, next_cursor}) so LLM knows what fields to extract.
Add explicit output schema documentation for diagnose (e.g. returns {status: string, checks: {health, database, server, all}, passed_checks: string[], failed_checks: {name, reason}}) so errors can be debugged.
Constrain search_docs 'library' parameter: either provide an enum of supported libraries (fastapi, pandas, numpy, torch, etc.) or document that invalid libraries return a helpful error listing available options.
Add recovery guidance to error responses: e.g. 'ChromaDB connection failed. Ensure CHROMA_HOST and CHROMA_PORT are set correctly. Check diagnose tool for health status.'
For search_docs with format='comprehensive', explicitly document 'returns top 20 results' in the tool description, not just in the parameter default.
Consider pagination for search_docs: add optional 'offset' and 'limit' parameters (or 'page' and 'page_size') to handle large result sets gracefully.
Document diagnose error conditions explicitly: what happens if ChromaDB is unreachable? What does a 'database' check failure return?

Spec posture evidence

Inferred effective spec: <=2025-11-25.

Relies on Logging (deprecated) - log to stderr or use OpenTelemetry

Score history

Overall score trend

↑ 18 points across a rubric change (v1 → v2)

55/100

Scored	Grade	Overall	Spec posture	Rubric
2026-09-22	D	55	<=2025-11-25	v2
2026-03-09	F	37	-	v1