Loading ToolBench
Local code knowledge graph — index repos, search semantically, and run multi-agent analysis pipelines
Three tools with complete schemas and reasonable descriptions. Naming follows verb_noun convention (search_, get_, list_). Descriptions are adequate (100-200 chars) but lack WHEN-to-use guidance and dependency hints. Parameters are typed and mostly described, but lack constraints (enums, ranges, patterns). Output schemas are not documented, LLMs cannot plan downstream calls. Error handling is minimal. No tool annotations (readOnlyHint, idempotentHint). search_codebase and get_file_context are well-scoped; list_repos is trivial. Overall: solid foundation, but missing LLM-optimization patterns.
Get the structural graph context for a specific file: its functions, classes, imports, and which other files import it. Use this after search_codebase to dive deeper into a specific file.
List all indexed repositories with their file counts, chunk counts, and local paths. Use this to understand what codebases are available to search.
Semantic search across all indexed code repositories. Returns the most relevant code chunks, documentation sections, or pipeline definitions matching the query. Use this to answer questions about how code works, find functions, or locate patterns across multiple repos.
Output schemas not documented. LLMs cannot infer what fields search_codebase returns (score, repo, file_path, start_line, end_line, chunk_type, language, name, content). Without documented return types, agents cannot plan chained calls or extract required data.
Parameter constraints missing. top_k accepts 1 - 20 but no min/max in schema. threshold accepts 0.0 - 1.0 but no bounds declared. LLMs may pass invalid values (top_k=1000, threshold=2.5) causing silent failures or API errors.
No tool annotations. All three tools are read-only and idempotent, but lack readOnlyHint and idempotentHint. Agents cannot distinguish safe tools from destructive ones, risking incorrect retry logic or permission assumptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Descriptions lack WHEN-to-use guidance. search_codebase says 'Use this to answer questions about how code works' but does not explain when to call it vs get_file_context, or what prerequisites exist (indexed repos required). Agents must guess intent.
Error handling is minimal. _search_codebase returns 'No results found above the threshold' as a string, not structured error. _get_file_context returns plain text errors. No recovery guidance (e.g. 'Try lowering threshold' or 'Run graphify graph --all first'). Agents cannot self-correct.