Loading ToolBench
The deterministic verification layer for AI coding agents. Catch invented APIs, undefined symbols, and wrong calls before your agent moves on.
HonestCode provides 18 tools with mostly complete schemas and descriptions. Naming is verb-forward and clear (scan_file, verify_file, check_symbol, execute_code). Descriptions average ~120 chars, meeting the 10 - 1024 char baseline. All tools have input schemas with type definitions. However, output schemas are not documented in the visible code, tool responses are described in docstrings but not formally specified in JSON Schema. Error handling is minimal; tools do not guide recovery or categorize errors. Parameter descriptions are present but often terse (e.g., 'Absolute path to the Python file' lacks context on when to call index() first). No tool annotations (readOnlyHint, destructiveHint) despite clear risk profiles (execute_code is WRITE, others READ_ONLY). Missing pagination guidance for tools returning lists (find_dead_code, find_similar_code, explore_call_graph). Overall: solid foundation, but lacks production-grade error guidance and output schema documentation.
Find files (especially tests) that may be affected by a git diff.
Verify a library API call is correct, e.g. pd.read_exel -> read_excel.
Check whether a symbol is actually defined in the project.
For debugging: choose which tool a natural-language query maps to.
Load dependency APIs declared in requirements.txt / pyproject.toml so library calls can be validated.
Execute code in a sandbox and return stdout/stderr plus structured error analysis with fix suggestions.
Explore callers and callees of a symbol across the project.
Output schemas not documented. Tool responses are described in docstrings but not formally specified in JSON Schema. LLMs cannot reliably parse or chain results without explicit output type definitions.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). execute_code is marked WRITE but lacks destructiveHint annotation. Agents cannot determine which tools are safe to retry or have side effects.
Error handling lacks recovery guidance. Tools do not return actionable error messages or categorize errors as retryable vs. fatal. A failed symbol lookup returns nothing; should suggest 'Try index() first' or 'Call search_code() to find the symbol.'
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | <=2025-11-25 | v2 |
Compute the blast radius of a symbol: transitively impacted symbols/files.
Find symbols that appear to be unused in the indexed project.
Find function-level code clones across the project.
Return statistics about the currently indexed project.
Build (or reuse) the symbol index for a project directory. Call this once before other tools. Subsequent calls return a cached index unless force_rebuild=True.
Load (and cache) API signatures for a specific installed package.
Scan a file for potential hallucinations: undefined symbols, missing imports, and incorrect API calls.
Search project source code with a regex pattern.
Stop the background file watcher, if any.
Run a lightweight AST-based structural check on code.
Verify a file and return structured evidence for the agent. Returns `status`, `findings` with `evidence`, and a `text` summary.
Parameter descriptions are terse and lack context. 'Absolute path to the Python file' does not explain when to call index() first or what happens if the file is not in the indexed project. Descriptions should be 50 - 200 chars with actionable context.
Pagination not documented for list-returning tools. find_dead_code, find_similar_code, explore_call_graph may return large result sets but lack limit/offset parameters or guidance on result capping. Large results risk context window exhaustion.