Loading ToolBench
Python implementation to spawn ephemeral Model Context Protocol (MCP) servers using the kubernetes API.
Server has 7 tools with adequate naming and generally complete schemas, but descriptions and parameter quality vary significantly. All tools use verb_noun naming (list_, create_, delete_, get_, mount_, remove_). Input schemas are present with type information for most parameters. However, descriptions for some tools lack depth about when to use them vs. similar tools, and several parameters have minimal or generic descriptions. Tools like mount_mcp_server and remove_mcp_server_mount have particularly thin documentation. Output schemas are not explicitly documented in the visible code, return types are inferred from docstrings but not formally specified. Error handling guidance is absent. Parameter descriptions generally meet minimum standards but some lack constraint information (e.g., create_mcp_server's 'env' parameter lacks detail on what environment variables are valid or required).
Create a new ephemeral MCP server in Kubernetes. Spawns a new MCP server instance as a Kubernetes Job with the specified configuration. The server runs in an isolated environment and can be customized with different runtime executors, packages, arguments, and environment variables. Args: runtime_exec: The executor to use for running the MCP server (e.g., 'uvx' for Python packages, 'npx' for Node.js packages, 'docker' for container images). runtime_mcp: The MCP package or image to run (e.g., 'mcp-server-sqlite' for a Python-based SQLite MCP server, or '@modelcontextprotocol/server-filesystem' for a Node.js filesystem server). runtime_args: Optional arguments to pass to the MCP server at runtime. These are appended to the command line when starting the server. env: Optional dictionary of environment variables to set in the server's container. Useful for configuration, authentication tokens, or feature flags. wait_for_ready: If True, this call will block until the Kubernetes Job is in a ready state. If False, returns immediately after submitting the Job. Returns: An EphemeralMcpServer object containing the created server's configuration, pod name, status, and connection details. Example usage: To create a filesystem MCP server: create_mcp_server( runtime_exec='npx', runtime_mcp='@modelcontextprotocol/server-filesystem', runtime_args='/tmp', wait_for_ready=True )
Delete an ephemeral MCP server and clean up its resources. Terminates a running MCP server instance and removes its associated Kubernetes resources including the Job, Pod, and any exposed Services. This is important for cleaning up resources when an MCP server is no longer needed. Args: job_name: The name of the pod/server to delete. This is returned when creating a server and can be retrieved using list_mcp_servers(). wait_for_deletion: If True, this call will block until the Kubernetes resources are fully deleted and removed from the cluster. If False, initiates deletion and returns immediately. Returns: An EphemeralMcpServer object containing the deleted server's final state and metadata before removal. Example usage: After retrieving the list of servers, delete a specific one by its pod name: delete_mcp_server(job_name='mcp-server-xyz123', wait_for_deletion=True)
mount_mcp_server and remove_mcp_server_mount lack substantive descriptions. 'Mount a remote MCP server over SSE' (53 chars) and 'Remove the mount of an MCP server' (34 chars) do not explain WHEN to use these tools, what SSE means in context, or their relationship to create_mcp_server/delete_mcp_server.
Output schemas are not formally documented. Docstrings mention return types (e.g., 'EphemeralMcpServer objects') but there is no explicit JSON Schema definition visible in the code for any tool return value. LLMs cannot parse unstructured docstring returns, they need formal schema.
Parameter 'env' in create_mcp_server lacks constraint guidance. Description says 'Optional dictionary of environment variables' but does not specify allowed keys, values, or whether certain variables are forbidden (e.g., secrets). LLMs may pass invalid or unsafe values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 66 | 2026-07-28+ | v2 |
| 2026-03-09 | C | 64 | - | v1 |
Get the detailed Kubernetes status of an MCP server. Retrieves the underlying Kubernetes Job status for a specific MCP server instance. This provides low-level details about the Job's execution state, including conditions, start/completion times, and any failure information. Args: job_name: The name of the pod/server to check. This identifier is returned when creating a server and can be retrieved using list_mcp_servers(). Returns: A Kubernetes V1Job object containing the Job's complete status information, or None if the Job is not found. The Job status includes fields such as: - active: number of active pods - succeeded: number of succeeded pods - failed: number of failed pods - conditions: detailed state information - start_time and completion_time Example usage: Use this to debug issues with a server or to check if a Job has completed successfully: get_mcp_server_status(job_name='mcp-server-xyz123')
List all currently running ephemeral MCP servers. Retrieves information about all MCP server instances that are currently managed by this controller. Each server entry includes its pod name, configuration details, current status, creation timestamp, and connection information if available. Returns: A list of EphemeralMcpServer objects representing all active MCP servers. Each object contains the server's configuration, runtime status, and metadata. Example usage: Call this tool to see what MCP servers are currently running before creating new ones or to check the status of existing servers.
List all available preset MCP server configurations. Returns a list of pre-configured MCP server templates that can be used as examples or starting points for creating new ephemeral MCP servers. Each preset includes the runtime executor (e.g., 'uvx', 'npx'), the MCP package to install, optional runtime arguments, and environment variables. Returns: A list of EphemeralMcpServerConfig objects containing preset configurations. Each config specifies how to run a specific MCP server package. Example usage: Use this to discover available MCP server configurations before creating one.
Mount a remote MCP server over SSE. Args: job_name: The name of the pod that is running the remote MCP server. name: The name of the proxy server.
Remove the mount of an MCP server. Args: name: The name of the server to remove. If None, all mounted servers with a prefix will be removed.
No error handling guidance. None of the tool descriptions mention what happens on failure, what errors are retryable, or how to recover. Example: create_mcp_server with wait_for_ready=true could timeout, but no guidance on retry behavior.
Parameter descriptions lack specificity on valid values. 'runtime_exec' accepts 'uvx', 'npx', 'docker' but the description uses 'e.g.' which suggests these are examples, not an enum. Should either be an enum type or explicitly state 'must be one of: uvx, npx, docker'.
mount_mcp_server parameter 'name' has description 'The name of the proxy server' but is optional with default=null. Unclear when this should be set vs. auto-generated. remove_mcp_server_mount's 'name' description mentions 'If None, all mounted servers with a prefix will be removed', wildcard behavior that could accidentally unmount unintended servers.