Loading ToolBench
Local semantic search over a folder of markdown. Uses hybrid BM25 + vector search with Reciprocal Rank Fusion when a vector index and embedding model are available, falling back to BM25-only and in-memory keyword search when services are unavailable.
Three tools with clear verb-noun naming (search, list_sections, get_section) and solid descriptions (194 - 250 chars, within baseline 34 - 392). All tools have input schemas with types and descriptions. However, output schemas are undocumented, responses are free-text strings rather than structured objects, forcing LLMs to parse unstructured text. Parameter descriptions are adequate but lack format constraints (e.g., top_k max=10 is enforced in code, not schema). Error handling is minimal, no recovery guidance or actionable error messages. No tool annotations (readOnlyHint, idempotentHint). Missing pagination for list_sections despite potentially large corpus. Overall: functional but below production baseline for agent-facing tools.
Get one section by file name and heading, with a fuzzy fallback.
List every indexed file and its section headings. A structured overview of what the corpus covers, useful before searching.
Search the corpus for relevant sections. Uses hybrid BM25 + vector search with Reciprocal Rank Fusion when a vector index and the embedding model are available, falling back to BM25-only, and to an in-memory keyword index when no LanceDB index exists. Returns full text for the top 3 results and a preview for the rest.
Output schemas undocumented. All three tools return free-text strings (str) with no structured schema. LLMs cannot parse results reliably or chain downstream tool calls. Responses should be JSON objects with typed fields (e.g., {results: [{file, heading, text, score}], mode, total}).
No tool annotations. Tools lack readOnlyHint (all are read-only), idempotentHint (search and list_sections are idempotent), or destructiveHint. Agents cannot infer safety properties without explicit hints.
Parameter constraints not in schema. top_k has max=10 enforced in code (line: top_k = min(max(top_k, 1), 10)) but not declared in input schema. LLMs may pass invalid values; validation happens at runtime with no guidance.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 67 | 2026-07-28+ | v2 |
No pagination for list_sections. Tool returns all sections as a single string. Large corpora will produce massive responses, exhausting context windows. Should support limit/offset and return structured results.
Error messages lack recovery guidance. get_section returns 'File not found: {file}' and 'Heading not found' but does not suggest alternatives or next steps. Should return available files/headings to enable agent self-correction.