Loading ToolBench
A safe-by-default remote agent runtime with stateless MCP, external OAuth resource-server auth, durable process handles, and optional CodeAct
Jeopsok demonstrates solid tool definition quality with comprehensive schemas, clear descriptions, and proper tool annotations. All 17 tools have explicit input schemas with type definitions and parameter descriptions. Tool names follow verb_noun conventions (python_session_create, exec_command, read_file). Descriptions are substantive (100-300 chars typical), explaining WHAT the tool does and WHEN to use it. However, some descriptions lack actionable error guidance, and output schemas are not formally documented. Tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) are present and correctly applied. Parameter constraints are well-defined (enums for runMode, signal; numeric bounds for timeouts). Risk classifications are explicit. Main gaps: no documented output schemas, limited error recovery guidance, and some parameter descriptions could be more prescriptive about format/constraints.
Read a permitted file chunk as base64. Continue with nextOffset until eof=true.
Run a shell command on the host. If it is still running after yieldTimeMs, the call returns a process sessionId that can be polled with read_process.
List a permitted directory. Recursive listing does not follow directory symlinks.
List running and recently completed process sessions.
Execute Python in a persistent full-profile session. Variables persist between calls. The host helper exposes host.files, host.process, and host.system. This is not a sandbox.
Inspect persistent Python state without resending it. Expressions are restricted to read-only indexing, slicing, comparisons, arithmetic, and a small set of pure builtins.
Output schemas not formally documented. Tools return results but LLMs cannot see expected field structure, types, or pagination info. Forces LLMs to infer output shape from examples.
Error handling lacks recovery guidance. Descriptions state what tools do but do not explain what to do if they fail (e.g., 'session not found' → try list_processes; 'permission denied' → check access policy).
Parameter descriptions lack prescriptive format constraints. E.g., 'code' in python_action says 'Python code to execute' but does not specify encoding, max length, or forbidden constructs. 'expression' in python_inspect says 'read-only' but does not list allowed builtins.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 79 | 2026-07-28+ | v2 |
Terminate a persistent CodeAct Python session and release its in-memory state.
Create a persistent Python worker for multi-step full-profile host work. Client input cannot enable unattended mode. Only server-owned MCP_RUN_MODE=unattended may remove interactive action/process-call budgets; otherwise sessions are interactive.
Read a bounded permitted file chunk as UTF-8 or base64. Continue with nextOffset until eof=true.
Poll a managed process for output and terminal state. Pass previous nextSeq as afterSeq for only newer output.
Permanently remove a permitted file or directory. There is no trash/recycle-bin layer.
Perform exact UTF-8 text replacement inside permitted roots. By default exactly one occurrence must exist.
Return metadata for a permitted file, directory, or symbolic link.
Send a signal to a managed process tree. SIGTERM escalates to SIGKILL after graceMs if necessary.
Write a base64 file chunk inside permitted roots at an exact byte offset.
Create, overwrite, or append content inside permitted roots.
Write text to an existing process session, optionally close stdin, then return new output.
list_processes and list_directory lack pagination parameters (limit, offset, cursor). Large result sets could exhaust context window. Descriptions do not mention result limits or truncation behavior.
Destructive tools (remove_path, terminate_process) lack confirmation/dry-run patterns. Agents can permanently delete files or kill processes without a safety gate. No mention of undo or recovery options.