Loading ToolBench
CodeGrok is a STDIO-only semantic code search tool with 4 tools. While it has detailed descriptions and reasonable naming, there are significant gaps in schema completeness, parameter documentation, and output structure definition. Tool names are action-verb based (learn, get_sources, get_stats, list_supported_languages), which is good. However, the code excerpt provided does not show complete input schema validation or output schema documentation. The 'learn' tool is the most complete with detailed description and input parameters, but get_sources lacks detail on output structure. Error handling and recovery guidance are not evident in the provided code. The server follows fastmcp framework conventions but lacks production-grade parameter validation and output schema documentation.
Semantic search for code
Get indexing statistics
Index a codebase for semantic search. REQUIRED FIRST STEP before using any other tool. Modes: - auto (default): Smart detection. If index exists, updates incrementally. If new, does full index. - full: Force complete re-index (destroys existing index). - load_only: Just load existing index without any indexing. Creates a .codegrok/ folder in the codebase directory.
List supported file extensions
Output schemas not documented. LLMs cannot plan downstream operations or extract needed fields from get_sources, get_stats, or list_supported_languages responses.
Tool descriptions for get_sources, get_stats, and list_supported_languages are too brief (under 50 chars). They lack context on WHEN to use the tool, what it returns, or how it relates to other tools. Baseline: 194 chars average for A+ tools.
No visible error handling or recovery guidance in tool implementations. Error responses should tell the LLM what to do next (e.g., 'Index not found. Call learn() first.').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 39 | - | v1 |
Parameter descriptions incomplete. The 'question' parameter in get_sources lacks guidance on what constitutes a good query. The 'mode' parameter in learn lists values but does not explain consequences (e.g., 'full' destroys existing index).
get_sources returns unspecified output. No documented schema for the response structure (e.g., list of code snippets with file paths, line numbers, relevance scores). Without this, agents cannot chain calls or extract needed data.
learn tool accepts 'file_extensions' parameter with default=null but no guidance on what happens when null is passed vs. an empty list vs. specific extensions. Parameter dependency unclear.