Loading ToolBench
A Python-based MCP server providing file system and command execution capabilities via FastAPI with WebSocket support
AgentBox exposes 8 file/command manipulation tools with basic parameter schemas but significant quality gaps. All tools have names starting with action verbs (list_, read_, write_, move_, delete_, execute_) and documented parameters with types. However, descriptions are uniformly terse (10-40 chars), parameter descriptions lack expected format/constraint details, output schemas are not documented, and there is no error handling or recovery guidance. The tool set includes high-risk operations (delete_file_or_dir, execute_command) with no confirmation mechanisms or permission checks. Most tools average 34-50 chars for descriptions, falling below the 50-200 char LLM-optimized baseline from production tools. Per the rubric baseline of 194 chars average for A+ tools, this server's descriptions are 50-75% of optimal. No enumeration constraints are used despite several parameters accepting limited values (recursive: true/false). The execute_command tool is particularly concerning: it accepts arbitrary shell commands with no injection protection, validation, or audit logging.
Delete a path or file.
Execute a shell command and return the output.
List all files in a directory.
Move a path or file.
Read the content of a file. Returns base64 encoded content.
Read the content of a file. Returns plain text content.
Write content to a file. Content should be base64 encoded.
Tool descriptions are uniformly terse (25-40 chars), 80% below the 194-char average for production A+ tools. Descriptions lack context on WHEN to use each tool, prerequisites, or return value expectations. Examples: 'List all files in a directory' (34 chars) vs 'Use this to list files in a directory and iterate through results; returns a paginated list up to 50 items. Useful for discovery before read operations.' (150+ chars).
No output schemas documented. Tools return raw types (list[str], str, bytes, None, CommandResult) without explaining what fields agents should expect. The CommandResult dataclass for execute_command is defined but its fields (stdout, stderr, returncode) are not described in tool documentation. LLMs cannot plan downstream tool chains without knowing what data they'll receive.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 53 | - | v1 |
Write content to a file. Content should be plain text.
execute_command accepts arbitrary shell commands with zero input validation, injection protection, or audit logging. No restrictions on dangerous patterns like 'rm -rf /', 'curl $(malicious_url)', or pipe chains. Shell=True flag enables command injection via argument interpolation. This violates pattern:tool-gateway and pattern:secret-injection security requirements.
Destructive operations (delete_file_or_dir, move_file_or_dir, write_file_* with overwrite) lack confirmation/dry-run support. An LLM planning to 'delete all backup files' could end up removing the wrong directory. No error guidance on recovery (e.g. 'Deletion is irreversible. Confirm: yes/no').
Parameter descriptions lack format/constraint details required by the rubric. Examples: 'directory' param in list_files has no guidance on absolute vs relative paths, symlink handling, or error cases. 'offset' and 'length' in read_file_text lack byte-size limits or clarification of behavior at EOF. 'command' in execute_command has no whitelist, injection protection guidance, or timeout documentation.
No error handling or recovery guidance. Tools invoke raw OS operations (os.listdir, open(), shutil.move, subprocess.run) without try/catch blocks or actionable error messages. If a file is not found, the LLM receives a raw FileNotFoundError exception. If move_file_or_dir fails due to permission denied, no guidance on next steps.
list_files returns unbounded results. If a directory contains 10,000 files, the tool returns all of them, bloating the response and context window. No maximum or pagination guidance in description. Best practice: cap at 20-50 items, add total_count, support offset/limit.
Security: No permission checks, rate limiting, or audit logging. Any agent can read/write/delete arbitrary files or execute any command. No scope declarations (e.g. 'read:/home/user, write:/tmp'). No validation that operations stay within intended boundaries (e.g. sandbox directories).
Tool composition issue: read_file_text and read_file_bytes do the same thing (read a file), differing only in encoding. Per the rubric, avoid multiple tools that do the same thing differently. Consider a single read_file tool with an optional encoding parameter (text|base64).