Loading ToolBench
A server for listing, searching, and retrieving documents curated exclusively for the project. If you want to know about project-specific knowledge or rules, you should first use the `overview` tool on this server to acquire the knowledge you need.
The Librarian server has well-structured tool definitions with clear input schemas using Zod validation. Most tools follow the verb_noun naming convention and include descriptions. However, there are significant gaps: (1) parameter descriptions lack depth and guidance for LLM reasoning (e.g., 'The directory path to list documents from' does not explain when to use depth=-1 vs a specific number); (2) output schemas are not formally documented in the tool definitions, only formatting functions exist in util.ts, making it unclear what LLMs should expect; (3) error handling is present but minimal, lacking actionable recovery guidance; (4) descriptions do not explain prerequisites or inter-tool dependencies (e.g., 'overview' should be called first, but tools don't cross-reference this); (5) the knowledge structuring session tools introduce state management (sessionToken) which complicates agent planning and is poorly documented. The server does well on naming consistency (listDocuments, searchDocuments, getDocuments, listTags, overview follow clear patterns) and all tools have descriptions. Schemas are well-typed via Zod, but the rubric requires OUTPUT schemas to be documented for the agent, not just input schemas.
Retrieves documents specified by their file paths. Since each document is small enough, please try to get as many documents as possible at once.
Instructs you to start a knowledge structuring session. Only use this tool when the user explicitly asks for it.
Ends a knowledge structuring session.
Displays the source document of the current session. Never use this tool unless you are explicitly instructed to do so in the previous tool response, as it requires a pre-generated session token.
Starts a knowledge structuring session. Never use this tool unless you are explicitly instructed to do so in the previous prompt, as it requires a pre-generated session token.
Writes the extracted section to the file. Never use this tool unless you are explicitly instructed to do so in the previous tool response, as it requires a pre-generated session token. Please specify as many sections as possible at once.
Output schemas are not documented. While input schemas are clear via Zod, the tool definitions do not include output/response schemas. LLMs cannot plan downstream tool calls or extract structured data without knowing what fields to expect. E.g., listDocuments and searchDocuments format results via formatDocumentList() but the schema for that output is not visible in the tool definition.
Parameter descriptions lack actionable context. E.g., 'Maximum directory depth to traverse (-1 for infinite)' does not explain WHEN to use -1 vs a specific number, or what performance implications exist. Descriptions should guide LLM reasoning, not just repeat the parameter name.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 53 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 53 | - | v1 |
Lists documents in the specified directory (defaults to the root dir), optionally filtering by tags and including contents.
Lists all tags in the specified directory (defaults to the root dir).
Returns a complete overview of all documents and tags in the library. You should first use this tool to see the structure of the library before using any other tools.
Searches documents in the specified directory (defaults to the root dir), optionally filtering by tags and including contents.
Knowledge structuring session tools introduce stateful interaction (sessionToken parameter) without explaining prerequisites or error recovery. The descriptions say 'Never use this tool unless explicitly instructed' but do not explain when an agent SHOULD use it, what a valid sessionToken looks like, or what errors indicate the token is stale. This breaks LLM composability.
Error handling is minimal. Try-catch blocks in server.ts catch errors and return a generic message ('Failed to [action]: <message>') with isError=true. This does not guide recovery: is the error retryable? Should the agent lookup a different parameter? Is this a fatal service outage? Errors should be categorized and include actionable next steps.
No inter-tool guidance or discovery hints. The overview tool description says 'You should first use this tool to see the structure of the library before using any other tools', good guidance, but other tools do not cross-reference this, and the description does not explain what structure overview reveals or what subsequent tools it enables.
No pagination or result limits documented. listDocuments, searchDocuments, and listTags return arrays. If a library contains thousands of documents, returned results could exhaust token budgets. Descriptions should state: 'Results are limited to 100 items per call; use offset/limit parameters for pagination' (or equivalent). Currently, no limits are visible.
searchDocuments accepts 'mode' enum with values 'string' and 'regex' but the description does not explain how to construct valid regex queries or what common regex patterns work. The note 'The string mode does not support OR/AND/NOT operators. To search for multiple words, use regex mode' hints at complexity but does not teach it.
knowledgeStructuring and knowledgeStructuringSession.* tools are stateful and async. The knowledgeStructure tool initiates a session and returns a response (presumably containing a sessionToken), but the description does not state that LLMs MUST extract and pass that token to the next call. This silent dependency breaks agent reasoning.