Loading ToolBench
The vast-rag server defines 3 read-only tools with complete JSON schemas and reasonable descriptions. Tool naming follows verb-noun convention (search_docs, list_collections, get_document). However, descriptions are formulaic and lack depth about when/why to use each tool. Parameter descriptions are present but minimal. Output schemas are not explicitly documented in code. The server is functional but lacks the polish and LLM-optimized guidance expected of production tools. No error handling patterns visible; no indication of how tools fail or what recovery looks like.
Retrieve metadata and content for a specific indexed document by its source filename.
List all document collections with their document counts. Shows how many chunks are indexed in each category.
Search indexed documentation using semantic similarity. Returns the most relevant document chunks for a query.
Incomplete tool descriptions lack actionable guidance. Descriptions state WHAT tools do but not WHEN or WHY to use them, making LLM selection ambiguous when multiple retrieval tools exist.
Output schemas not documented in source code. The Python implementation returns dicts (e.g., search_docs returns list[dict] with 'text', 'source', 'score', 'category', 'page', 'section' fields) but no explicit output schema is visible in the tool registration. LLMs cannot reliably plan downstream operations without knowing expected response structure.
No error handling or recovery guidance visible. The code shows no error handling patterns (try/catch blocks, validation, or recovery hints in error messages). If search_docs fails or returns no results, the LLM has no guidance on what to do next.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | <=2025-11-25 | v2 |
Parameter 'n_results' constraint unclear for LLM. Schema declares minimum=1, maximum=20 with default=5, but description does not explain the range or why max is 20. An LLM might attempt to request 100 results and fail.
list_collections has no parameters but description does not explain when to call it (discovery before search). Missing dependency hint: 'Call this first to see which categories are available before searching.'
No tool composition guidance. The source code shows three separate tools (search_docs, list_collections, get_document) but provides no hint about which combinations are idiomatic. E.g., should an LLM always list_collections first? Can get_document substitute for search when exact filenames are known?