Loading ToolBench
Local-first engineering agent with proof receipts, project memory provenance, MCP, skills, and terminal TUI.
soba-agent provides 11 tools with basic schemas and descriptions, but falls short of production quality. Most tools have descriptions (8-50 chars, below the 50-200 char baseline for LLM optimization) and input schemas with type definitions. However, parameter descriptions are sparse or missing entirely. For example, 'bash' tool lacks timeout description detail; 'edit' tool's 'changes' array parameter has no schema for array items. Output schemas are not documented. Error handling is minimal, no recovery guidance, no categorization of retryable vs fatal errors. Tool composition is reasonable (read, write, bash, ls, search_files, inspect_file form a coherent file-system toolkit), but descriptions do not explain WHEN to use each tool vs alternatives (e.g., when to use inspect_file vs read). Security concerns: bash tool executes arbitrary commands with no validation or permission gates. No audit logging visible. Naming is verb-first and clear (read, write, bash, ls, search_files, inspect_file, checkpoint, ask_user, activate_skill, deactivate_skill), meeting baseline conventions.
Activate a skill extension
Ask user for clarification or input
Execute bash commands
Create project checkpoint with work plan state
Deactivate a skill extension
Edit file with scoped changes
Inspect file with line range context
List directory contents
bash tool executes arbitrary commands with no input validation, sanitization, or permission gates. No dry-run or confirmation step for destructive operations. Agents can trivially invoke rm -rf / or exfiltrate secrets.
Parameter descriptions are missing or trivial. 'edit' tool's 'changes' parameter is described as 'Array of edit operations' with no schema for array items (what fields? what types?). 'checkpoint' parameters lack guidance on valid values for 'kind' and 'reason'. LLMs cannot infer correct usage.
Tool descriptions are too short (35-55 chars) to guide LLM selection. Baseline is 50-200 chars. 'bash' is 'Execute bash commands', does not explain when to use bash vs other tools, what risks exist, or what output format to expect. 'checkpoint' is 'Create project checkpoint with work plan state', vague on what 'checkpoint' means or when to call it.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Read text file content
Search files by query pattern
Write text file content
Output schemas are not documented. Tools return structured data (e.g., 'edit' returns success/failure, 'search_files' returns matches), but the response format is not specified in tool definitions. LLMs cannot plan downstream calls or extract required fields.
No error handling guidance. Tools do not document what errors are retryable, what errors require user input, or what recovery steps are available. A bash command timeout or file-not-found error returns raw output with no actionable next step for the LLM.