Loading ToolBench
Two tools with basic schemas and descriptions, but significant gaps in parameter documentation, error handling, and output structure. Tool names lack action verbs (build, run are generic). Descriptions exist but are minimal (under 100 chars). Parameter descriptions are present but incomplete, 'args' lacks format guidance, 'timeout_seconds' is documented as 'not used', creating confusion. No output schema documented. No error recovery guidance. No idempotency or safety markers. Security concern: subprocess execution without input sanitization against command injection.
Build go Project in project_dir param and reply with success/fail
Run a Go executable from the specified project directory for max 5 seconds
Generic tool names lack action verbs. 'build' and 'run' are vague, LLMs cannot distinguish intent. Recommend 'build_go_project' and 'run_go_executable' to clarify scope and reduce ambiguity.
Parameter 'timeout_seconds' in run() is documented as 'not used, always exits after 5 sec', creating a contract violation. LLMs will pass this parameter expecting it to work. Either honor the parameter or remove it and hardcode the 5-second limit.
No output schema documented. Both tools return Tuple[str, bool] but LLMs cannot parse this. Document: { success: boolean, output: string, executables?: string[] } to enable downstream tool chaining.
Subprocess execution in build() and run() lacks input sanitization. 'args' parameter is split via shlex but project_dir is passed directly to subprocess.run(). Vulnerable to path traversal and command injection. Validate project_dir against a whitelist or jail.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | <=2025-11-25 | v2 |
Error messages are unstructured strings. No recovery guidance. E.g., 'Error: Directory does not exist' does not tell LLM what to do next. Adopt pattern: { error: string, recovery_hint: string, retryable: boolean }.