Loading ToolBench
A monorepo application with API, web, and worker services that provides RAG (Retrieval-Augmented Generation) chat capabilities with MCP (Model Context Protocol) integration for code search and documentation retrieval
ConvergeKit exposes three read-only codebase-search tools with clear verb-noun naming (searchDocs, readFile, getStructure). All three have descriptions (194 - 250 chars, within baseline 34 - 392 range) and input schemas with types. However, parameter descriptions are sparse or missing entirely: searchDocs lacks descriptions for 'chunkType' enum values; readFile's 'path' param has a description but no format/length constraints; getStructure's 'pathPrefix' lacks guidance on expected format. Output schemas are not documented, LLMs cannot predict what fields searchDocs returns or how pagination works. Error handling is absent from visible code. No tool annotations (readOnlyHint, idempotentHint) despite all three being read-only and idempotent. Transport is unknown, capping protocol readiness.
Get a bounded file and folder structure listing for the repository. Optionally filter by a path prefix to explore a specific directory.
Read raw content of a file from the repository by its path. Large files are returned as a bounded preview with truncation metadata; use searchDocs for targeted chunks.
Search the codebase for relevant code snippets and documentation using hybrid semantic + keyword search. Returns the top matching chunks with file path and line numbers for source attribution.
Output schemas not documented. LLMs cannot predict what fields searchDocs returns (e.g., does it return file_path, line_number, snippet, score?). Without documented return types, agents cannot chain tools or extract required data.
Parameter descriptions incomplete. searchDocs 'chunkType' enum lacks descriptions for each value (function, class, method, module, block, comment). readFile 'path' lacks format constraints (relative vs absolute, allowed characters). getStructure 'pathPrefix' lacks guidance on format or examples.
No tool annotations despite all three tools being read-only and idempotent. Missing readOnlyHint and idempotentHint annotations prevent clients from optimizing caching and retry logic.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
No error handling guidance. If searchDocs finds no results, readFile hits a 404, or getStructure times out, the visible code provides no recovery hints. LLMs need actionable error messages.
searchDocs 'limit' parameter (1 - 20, default 5) lacks guidance on when to increase. LLMs may not understand that higher limits increase latency or token cost. Add context: 'Higher limits return more results but increase response size and latency.'