Loading ToolBench
Production-ready agentic framework for building advanced AI agent systems and multi-agent orchestration with MCP support
The Promptise Foundry MCP server exhibits critical definition quality issues across the board. While 15 of 19 tools have descriptions ranging from moderate to good length (sandbox tools are well-documented at ~200-300 chars each), the meta-tools (create_tool, connect_mcp_server, spawn_process, etc.) have superficial descriptions that fail to guide LLM decision-making. More critically, NO input schemas are visible in the provided source code, only parameter names and type hints in docstrings. JSON Schema validation objects (required for MCP compliance) are not evident. Parameter descriptions exist but are inconsistent: sandbox tools describe constraints clearly ('timeout in seconds (default: 300)'), while meta-tools omit critical details (e.g., 'add_trigger' lists 7+ parameters with minimal guidance on which are mutually exclusive). The meta-tool parameter descriptions are generic and non-actionable, e.g., 'Trigger type: cron, event, message, webhook, file_watch, or a custom-registered type' lacks format/constraint details. Error handling guidance is absent; tools offer no recovery hints. The architecture introduces 6 meta-tools (modify_instructions, create_tool, connect_mcp_server, spawn_process, add_trigger, remove_trigger) that modify runtime behavior, these are extremely high-risk and should declare permissions, constraints, and rollback mechanisms, but do not. Per-tool analysis shows sandbox_* tools (1-5) are reasonably well-crafted (~65 each), but meta-tools (6-11) average ~20-30 due to vague parameter semantics and absent schemas. Overall composition is poor: 'create_tool' and 'modify_instructions' operate on the agent itself, violating separation of concerns and creating runaway-agent risks.
Add a trigger (cron, event, message, webhook, file_watch) dynamically (meta-tool in OPEN execution mode)
Check current token/cost budget (meta-tool in OPEN execution mode)
Check mission/goal state (meta-tool in OPEN execution mode)
Connect to an additional MCP server dynamically (meta-tool in OPEN execution mode)
Define a new Python function tool dynamically (meta-tool in OPEN execution mode)
Delete a memory entry (meta-tool in OPEN execution mode)
Retrieve a secret from environment or vault (meta-tool in OPEN execution mode)
No JSON Schema input definitions visible in source code. All 19 tools lack formal inputSchema declarations; only docstring type hints present. MCP requires explicit schemas for tool registration.
Meta-tools (modify_instructions, create_tool, connect_mcp_server, spawn_process, add_trigger) expose dangerous runtime mutation capabilities without permission gates, scope declarations, or confirmation mechanisms. These tools can fundamentally alter agent behavior and compromise security.
add_trigger parameter descriptions are vague and do not clarify mutual exclusivity. 'trigger_type' accepts 'cron, event, message, webhook, file_watch, or a custom-registered type' but does not document which parameters apply to each type (e.g., cron_expression only valid for cron triggers). LLM cannot safely construct valid requests.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
Introspect current tools, triggers, and identity (meta-tool in OPEN execution mode)
List all processes in the runtime (meta-tool in OPEN execution mode)
Modify system prompt / instructions for the agent (meta-tool in OPEN execution mode)
Remove a dynamically added trigger (meta-tool in OPEN execution mode)
Execute a shell command in the secure sandbox environment. The sandbox is an isolated Linux container where you can safely: - Run Python, Node.js, Rust, Go, or any other installed tools - Install packages (pip, npm, cargo, go get, etc.) - Create and modify files in /workspace - Run tests and experiments - Execute any CLI commands Examples: - sandbox_exec(command="python --version") - sandbox_exec(command="pip install requests && python script.py") - sandbox_exec(command="npm install express && node server.js", timeout=60) - sandbox_exec(command="ls -la /workspace") The command runs in /workspace by default. Output includes stdout, stderr, and exit code.
Install a package in the sandbox environment. Supports multiple package managers: - Python: pip install - Node.js: npm install -g - Rust: cargo install - Go: go install Examples: - sandbox_install_package(package="requests", tool="python") - sandbox_install_package(package="express", tool="node") - sandbox_install_package(package="ripgrep", tool="rust") Returns installation output or error.
List files in a directory inside the sandbox. Use this to see what files exist in the sandbox workspace. Examples: - sandbox_list_files(directory="/workspace") - sandbox_list_files(directory="/workspace/data") Returns list of file and directory names.
Read a file from the sandbox environment. Use this to read files you've created or downloaded in the sandbox. Examples: - sandbox_read_file(file_path="/workspace/script.py") - sandbox_read_file(file_path="/workspace/data/results.json") Returns the file contents as a string.
Write a file to the sandbox environment. Use this to create or overwrite files in the sandbox. Examples: - sandbox_write_file(file_path="/workspace/script.py", content="print('hello')") - sandbox_write_file(file_path="/workspace/config.json", content='{"key": "value"}') Returns success message or error.
Search long-term memory (meta-tool in OPEN execution mode)
Spawn a new agent process in the runtime (meta-tool in OPEN execution mode)
Store information in long-term memory (meta-tool in OPEN execution mode)
Meta-tools lack error handling guidance and recovery hints. If create_tool fails (syntax error, name collision, permission denied), the tool returns no actionable error message directing the LLM what to do next.
Memory tools (store_memory, search_memory, forget_memory) do not document output schema. What fields does search_memory return? What is the structure of a memory entry? LLMs cannot extract the 'memory_id' field if forget_memory requires it but search_memory does not return it.
create_tool parameter 'parameters' and 'python_code' descriptions are incomplete. 'JSON schema for tool parameters' does not explain format, required vs optional, or nested object handling. 'Python code for the tool function body' does not specify imports, dependencies, or exception handling requirements.
No pagination or result limits documented for list_processes, list_capabilities, search_memory. If these return hundreds of items, they will exhaust context windows. Baseline: limit to 20-50 items with next_cursor support.
spawn_process 'triggers' parameter is documented as 'array' with no schema. What is the structure of each trigger config dict? How does this relate to add_trigger? The lack of clarity forces the LLM to guess or fail.
Sandbox tools accept arbitrary command strings without validation guidance. sandbox_exec(command='rm -rf /') is technically valid but catastrophic. No constraints on dangerous patterns (rm, dd, format, etc.) are documented.
connect_mcp_server parameter 'transport' description lists 'streamable-http, sse' but does not explain when to use each, deprecation status of SSE, or what happens if the server URL is unreachable. Also: no validation of URL format or timeout behavior.