Loading ToolBench
The server has solid tool definitions with well-structured Pydantic input models and comprehensive parameter documentation. All 7 tools have descriptions and schemas visible in the code. However, there are notable gaps: output schemas are not formally documented in the code (only descriptions mention what is returned), error handling lacks actionable recovery guidance, and some parameter descriptions could be more precise about expected formats and constraints. The tool naming is clear and verb-based, which is a strength. The design reflects good understanding of AST/code analysis patterns but falls short of production-grade polish in output documentation and error classification.
Get the type hierarchy for a class. Returns base classes (upward inheritance) and known derived classes (downward inheritance).
Get a class outline or file outline. For classes, returns member signatures without bodies. For files, returns all top-level declarations.
Find all references and call sites for a symbol. Returns caller name, file, line number, and surrounding context.
Look up a C++ symbol by name. Returns signature, metadata, location, documentation, and optionally the full source body.
Trigger (re-)indexing of a C++ project. Indexes all source files in a directory or processes a compile_commands.json.
Keyword and semantic search across C++ symbols. Searches symbol names, signatures, doc comments, and parameter types. Ranks results by relevance using BM25.
Output schemas not formally documented. Tool descriptions mention what is returned (e.g., 'Returns signature, metadata, location, documentation') but the actual output JSON structure is not defined in code comments or a separate schema reference. This forces LLMs to infer response structure and risks field mapping errors.
Error handling lacks actionable recovery guidance. The code shows logging of errors and exceptions raised, but no structured error categorization (retryable vs. user-fixable vs. fatal) or recovery hints in error messages. E.g., if 'symbol not found', should the LLM call ast_search() first?
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 60 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 23 | - | v1 |
Get the current status of the AST index. Returns counts of indexed symbols, references, and files.
Parameter descriptions lack format/constraint details for some fields. E.g., 'name' parameters in ast_get_outline, ast_get_hierarchy accept 'class name or file path' but do not specify: What is the exact format for a file path? Is it relative or absolute? Does it need a file extension? LLMs need explicit format guidance.
ast_search 'kinds' parameter accepts an optional array of enums but does not clarify: what happens if an invalid kind is passed? Are they silently ignored, or does the tool return an error? This ambiguity can lead to silent filtering failures.
No pagination support documented for ast_search results. If a query matches thousands of symbols, the default limit of 10 may not be sufficient, but no next_cursor or offset mechanism is described for follow-up queries. Large result sets could truncate without a way to fetch more.
Tool composition clarity: ast_search returns results with 'relevance' ranking, but does not document whether returned symbol names can be directly passed to ast_get_symbol or if additional qualification is needed. Clear chaining guidance is missing.