Loading ToolBench
Support Python development with checkers and a scoped interpreter
Five tools with clear names and functional descriptions, but significant gaps in schema documentation, parameter descriptions, and error handling guidance. Tool names follow verb_noun convention well (run_python, check_python, fix_python, security_scan, list_installed_packages). Descriptions are present but lack depth on WHEN to use each tool and WHAT the LLM should expect downstream. Input schemas visible in the code show basic parameter typing (code: string, filename: string) but lack enum constraints, format specifications, or validation rules. Output schemas are partially visible but underdocumented in descriptions, the run_python return dict structure is clear in code but not formally documented as part of the tool definition metadata. Error handling is minimal; timeout errors and failures do not guide recovery. No parameter descriptions explain format, constraints, or dependencies. No tool indicates whether it modifies state (run_python clearly does; list_installed_packages is read-only but not explicitly marked). Parameter 'filename' has a default but no description of allowed formats. No tool includes actionable error messages, stderr output is returned raw without guidance on common failures.
Run linting and type checking on Python code.
Auto-fix Python code issues using ruff.
List all Python packages installed via pip.
Execute Python code in an isolated environment with security restrictions.
Run security scanning on Python code using Ruff's Bandit security rules.
Output schemas not formally documented. run_python, check_python, fix_python, and security_scan all return complex dicts (with stdout, stderr, returncode, session_dir, ruff_issues, etc.), but none of these response structures are described in tool metadata. LLMs cannot plan downstream calls or extract the right fields.
Parameter descriptions are missing or trivial. 'filename' appears in all tools with only default value documented, not format/constraints. No description of what 'code' parameter should contain (is it a full script, a snippet, raw bytes?). No mention of maximum code length or character restrictions.
Error handling provides no recovery guidance. Timeout errors return a message but no suggestion (e.g., 'Execution timeout (30 seconds exceeded). Try breaking the code into smaller functions or adding early exit conditions'). ImportError from blocked modules returns raw exception. stderr output is returned unparsed.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 48 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 44 | - | v1 |
State modification not declared. run_python and fix_python modify state (execute code, rewrite files) but descriptions do not say so. LLMs need to know which calls have irreversible consequences. check_python, security_scan, and list_installed_packages are read-only but not marked as such.
Session directory leakage. Tools return 'session_dir' in responses, exposing temporary paths to the LLM. This is an internal implementation detail that should be hidden; returning it invites agents to reference or manipulate the directory, creating security and cleanup issues.
list_installed_packages has empty input schema but no documentation of output structure. Code does not show the actual return format, is it a list of package names, a list of dicts with name/version, or raw pip output?
No bounds or enum constraints on parameters. 'filename' has no length limit or character restrictions. 'code' parameter unbounded, could accept megabytes. No way for LLM to know what values are valid.