Loading ToolBench
Self-hosted secure AI agent sandbox runtime with Docker container isolation, MCP protocol support, audit trails, and task management
VaultRun demonstrates solid tool design with 22 well-named, action-verb-prefixed tools (create_session, run_command, delete_file, etc.). All tools have descriptions (avg ~120 chars, within 10-1024 baseline). Input schemas are present and typed for all tools. However, several critical gaps reduce the score: (1) Parameter descriptions lack detail on constraints, ranges, and formats, e.g., 'timeout_seconds' has no min/max guidance; (2) Output schemas are not documented, LLMs cannot predict response structure; (3) Error handling is minimal, no recovery guidance or actionable error messages visible; (4) Some parameters use string enums (e.g., 'true'/'false' as strings) instead of proper boolean types; (5) Tool descriptions omit dependency hints and prerequisites. The server excels at naming clarity and basic schema presence but falls short on LLM-optimized descriptions and output documentation.
Cancel a working MCP Task. Prefer this over tasks/cancel when the host only supports tools.
Promote a file from the session workspace to the shared artifact registry. Artifacts are accessible across sessions and can be downloaded independently.
Create a new isolated sandbox session with a Docker container. Returns the session ID needed for subsequent operations. Always delete the session when done to free resources.
Save the current workspace state as a named snapshot archive. Snapshots can be used to checkpoint progress or restore state later.
Delete a file from the session workspace.
Stop and delete a sandbox session, freeing all resources. Call this when you are done with a session.
Output schemas not documented. LLMs cannot predict response structure for any tool (list_sessions, get_session, run_command, etc.). This forces agents to guess field names and types, increasing errors and wasting tokens on exploratory calls.
String-based boolean parameters (enum: ['true', 'false']) instead of proper boolean type. Affects 'network_enabled', 'async', 'confirm', 'approve'. LLMs may pass 'True', '1', or 'yes' instead of the exact string 'true', causing validation failures.
Numeric parameters lack min/max constraints. 'timeout_seconds', 'cpu_limit', 'memory_limit_mb', 'limit' have no documented ranges. LLMs may pass 0, negative, or absurdly large values (e.g., timeout_seconds=999999), causing API failures or resource exhaustion.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Get details of a specific command run including its stdout, stderr, exit code, and status.
Get details about a specific session including its status.
Get live CPU/memory/network stats for the session's container.
Poll an async MCP Task started with async=true on a tool (e.g. run_command). Returns status, progress, result/error, and any pending inputRequests. Prefer this over the tasks/get custom method when the host only supports tools.
List all shared artifacts available across sessions.
Query the immutable audit trail. Returns a chronological log of all actions (session creation, command execution, file uploads, etc.).
List all files in the session workspace, including files created by running commands.
List Docker images available on the VaultRun host. Requires master API key.
List all command runs for a session, showing their status, exit codes, and durations.
List all active sandbox sessions. Returns session IDs, images, status, and creation times.
List all workspace snapshots for a session.
Pull a Docker image from a registry. Requires master API key.
Read the content of a file from the session workspace. Use this to retrieve output files created by commands.
Execute a command inside a sandbox session. Returns stdout, stderr, exit code, and duration. The command runs as an unprivileged user inside an isolated container. For multi-step workflows, prefer uploading a script file and running it rather than chaining shell commands.
Update a working MCP Task: optional progress/message, or answer input_required (approve/decline confirm, or JSON input_responses). Prefer this over tasks/update when the host only supports tools.
Upload a file to the session workspace. The file is immediately available at the given path inside the container. Use this to upload scripts, data files, or configuration before running commands.
Error handling not visible in tool definitions. No recovery guidance, error classification, or actionable messages documented. Agents cannot determine if errors are retryable, user-fixable, or fatal.
Parameter descriptions lack format/constraint details. E.g., 'path' in upload_file/read_file has no guidance on absolute vs relative, allowed characters, or max length. 'image' in create_session/pull_image lacks validation rules (e.g., 'must be a valid Docker image reference').