Loading ToolBench
Search and read markdown documentation in the local corpus using TF-IDF retrieval
Single tool with adequate naming (search_docs) and basic schema, but critical gaps in parameter descriptions and output documentation. The tool description is present (71 chars) but lacks context on when to use it vs. alternatives. Parameter 'k' has a description but lacks constraints (min/max bounds). No output schema documented, LLMs cannot predict the structure of returned list[dict]. The resource tool (get_doc) is present but not registered as a tool in the MCP sense, so it does not appear in the tool list. Error handling is minimal, no guidance on what happens if a path is invalid or docs_dir is missing.
Search the documentation corpus with TF-IDF retrieval.
Output schema not documented. Tool returns list[dict] with fields {path, chunk_index, score, excerpt}, but LLMs have no formal schema to predict structure or plan downstream calls.
Parameter 'k' lacks numeric constraints. Description says 'default 5' but does not specify min/max bounds. Code enforces max(1, min(k, 20)) internally, but LLMs cannot see this, they may pass invalid values.
Tool description lacks context on when to use it. Does not explain what TF-IDF retrieval is, when to call search_docs vs. get_doc, or what the score field represents. LLMs cannot determine if this is the right tool for a given query.
No error handling guidance. If docs_dir is empty, search returns []. If query is empty, search returns []. LLMs receive silent failures with no recovery hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
Resource tool (get_doc) is not exposed as a tool in the MCP tool list. It is registered as a resource (doc://{path}), which is correct for MCP, but the server instructions mention 'get_doc' as if it were a tool. This creates ambiguity about how agents should access full documents.