Loading ToolBench
MCP-powered AI research assistant with RAG, LangGraph agents, and FastAPI. Searches papers, analyzes PDFs, performs RAG retrieval, and generates cited research reports.
ResearchPilot-MCP has 8 tools with reasonable naming (verb-noun pattern) and basic descriptions. However, critical gaps undermine quality: (1) Parameter descriptions are minimal or absent, most params lack detail on format, constraints, or valid ranges. (2) Output schemas are not documented anywhere in the provided code; LLMs cannot infer what fields to expect from tool responses. (3) Error handling is not visible, no recovery guidance, categorization, or actionable error messages. (4) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk distinctions (read_pdf, generate_report, export_* are WRITE operations). (5) Tool descriptions are generic and lack context on when to use each tool vs. alternatives. Naming is solid (search_papers, retrieve_context, verify_claim), but execution falls short of production standards.
Export a research report to Markdown format
Export a research report to PDF format
Generate a comprehensive research report on a topic using the LangGraph workflow
Extract text from a PDF file and optionally index it into the RAG vector store
Retrieve relevant context from ChromaDB for a given query using RAG
Search academic papers from arXiv and Semantic Scholar by query
Generate a summary of a document or text using an LLM
Verify a claim against indexed research papers using RAG and LLM reasoning
Output schemas not documented. LLMs cannot infer what fields tools return (e.g., does search_papers return [title, authors, url, abstract, arxiv_id]? Does retrieve_context return [text, source, score]?). This forces agents to guess and wastes tokens on trial-and-error.
Parameter descriptions are minimal or missing. 'max_results' lacks range (1 - 100? 1 - 1000?). 'top_k' has no guidance on typical values. 'paper_metadata' is vague, what fields does it accept? This violates the baseline that 100% of A+ tools have param descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 34 | <=2025-11-25 | v2 |
No tool annotations despite clear risk profile. read_pdf, generate_report, export_markdown, export_pdf are WRITE operations that modify state (index PDFs, create reports, write files). They should be marked with destructiveHint or idempotentHint so agents know retry safety. search_papers, retrieve_context, summarize_document, verify_claim are READ_ONLY but lack readOnlyHint annotation.
No error handling guidance visible. If search_papers returns 0 results, what should the agent do? If read_pdf fails on a corrupted file, should it retry or ask the user? If generate_report times out, is it retryable? Agents need categorization (retryable, user-fixable, fatal) and recovery hints.
Tool descriptions lack context on when to use each tool vs. alternatives. 'summarize_document' and 'verify_claim' both process text, what's the difference? When should an agent call summarize vs. verify? Descriptions should answer 'When should the LLM call this instead of a similar tool?'