Loading ToolBench
MCP server for searching arXiv papers, downloading PDFs, and performing RAG (Retrieval-Augmented Generation) on research papers using ChromaDB and Ollama embeddings
Server has 5 tools with basic HTTP FastAPI routing. All tools have descriptions (10-60 chars, mostly short), but parameter descriptions are minimal or absent. Input schemas are inferred from FastAPI route signatures rather than explicitly defined JSON Schema objects. No output schemas documented. Error handling exists (HTTPException) but lacks recovery guidance. Tool naming follows verb_noun pattern (get_papers, search_and_ingest, wipe_chroma, retrieve, clear_assets), but 'search_and_ingest' violates single-responsibility principle. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels. Parameter constraints (e.g., top_k range, query format) not documented. Destructive operations (wipe_chroma, clear_assets) lack confirmation patterns.
Delete the /assets directory
Fetch a list of 20 papers from arXiv
Retrieve relevant document chunks from ChromaDB for a given query
Fetch papers, download PDFs, and embed into ChromaDB
Wipe the ChromaDB vectorstore
search_and_ingest violates single-responsibility principle: fetches papers, downloads PDFs, and embeds into ChromaDB in one tool. Should split into separate tools (search_papers, ingest_papers) so agent can compose them independently.
No tool annotations despite clear risk levels. Tools marked READ_ONLY, WRITE, and DESTRUCTIVE in metadata but not exposed via MCP readOnlyHint/destructiveHint/idempotentHint. LLM cannot infer safety properties.
Parameter descriptions missing or trivial. 'query' param described as 'Search query string' (18 chars), no guidance on format, length, or examples. 'top_k' lacks range constraints (1-100?). 'wipe_db' lacks explanation of consequences.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | <=2025-11-25 | v2 |
No output schemas documented. Callers cannot predict response structure. search_and_ingest returns {papers, indexed_chunks, wipe_db} but this is inferred from code, not declared. get_papers returns list of {title, summary, published, authors}, undocumented.
Destructive operations (wipe_chroma, clear_assets) lack confirmation or dry-run patterns. Agent can irreversibly delete vectorstore or /assets directory without safeguards. No recovery guidance in error responses.