Loading ToolBench
Local-first AI coding agent with Ollama, automatic multi-model routing, MCP tools, skills, project memory, and human approval gates.
Codesmith has 8 tools with clear verb-based naming (list_dir, read_file, write_file, edit_file, run_command, update_plan, read_skill, spawn_agent). All tools have descriptions (10 - 200 chars, within baseline 194 avg). Input schemas are visible and typed. However, several tools lack parameter descriptions, output schemas are undocumented, and error handling guidance is minimal. The tool set is well-composed (single responsibility, good chaining via path parameters), but descriptions could be more LLM-optimized and parameter constraints are underspecified.
Replace an exact block of text in an existing file with new text. old_string must match the file's current content exactly and, unless replace_all is true, must be unique in the file.
List files and subdirectories at a path inside the project (relative to the project root).
Read a text file inside the project, with line numbers. Optionally a line range.
Load the full instructions for a skill by its exact name.
Run a shell command in the project directory with optional timeout and output truncation.
Spawn a read-only sub-agent to investigate a self-contained question or summarize a subsystem. The sub-agent can only call list_dir, read_file, and read_skill.
Output schemas undocumented. No tool describes what fields it returns or their types. LLMs cannot plan downstream calls or extract chaining IDs (e.g., what does read_file return? structured object or plain text?).
run_command lacks error guidance. Description does not explain what happens on timeout, non-zero exit, or command injection risk. No recovery hints for LLM.
update_plan and spawn_agent have vague descriptions. 'Create or update a multi-step plan' and 'Spawn a read-only sub-agent' lack context on when to use them vs. direct execution. No guidance on plan format or sub-agent capabilities.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Create or update a multi-step plan for the current task. Pass the entire updated list each time, marking steps as pending, in_progress, or completed.
Create a new file, or fully overwrite an existing one, with the given content.
edit_file's replace_all parameter lacks constraint documentation. Description does not specify behavior if old_string appears 0 times, or if new_string is empty. No guidance on idempotency.
run_command timeout_s and max_output_chars parameters lack range constraints. No min/max documented. LLM could pass negative values or absurd timeouts (e.g., 999999 seconds).