Loading ToolBench
API for managing repositories and MCP servers with testing, Docker deployment, and vector database integration
This MCP server has 27 tools across repository management, configuration, Docker integration, and MCP testing. While all tools have descriptions and visible schemas, the quality varies significantly. Many descriptions are generic or fail to explain WHY an agent should use them versus similar tools. Parameter descriptions are minimal or missing context about formats and constraints. Output schemas are not documented, only input schemas are visible. No error handling guidance is present. The tool set shows composition issues: multiple tools could be combined (e.g., create_repository and test_mcp_server are separate but commonly used together). Critical security concerns include parameters for 'api_key' and 'env' that could leak secrets. Most tools score in the 35 - 55 range, dragging the average down significantly.
Test multiple MCP servers in parallel. Tests multiple repositories concurrently, providing aggregate results and individual reports.
Check if a host port is available.
Clean up all running test containers. Stops and removes all containers created by the MCP testing system.
Create a new repository entry in the database.
Delete a repository from the database.
Disable automatic testing for newly added repositories.
Download the generated YAML configuration file.
Missing output schemas for all 27 tools. No documentation of what fields agents should expect in responses, forcing LLMs to guess structure and preventing reliable chaining between tools.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 58 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 37 | - | v1 |
Enable automatic testing for newly added repositories.
Fetch details for a repository including README, Dockerfile content, and GPT analysis.
Generate and save YAML (or JSON if extension is .json) configuration file.
Get the status of the automatic testing service.
Check if Qdrant collections exist and are accessible.
Check Docker connectivity status.
Generate JSON configuration for MCP servers.
Get current Qdrant connection status.
Get a repository by name.
Get all configuration settings.
Get the current status of the MCP testing system. Returns information about Docker connectivity, running test containers, and system resources.
Generate YAML configuration for MCP servers.
Simple health check for the MCP testing API.
Initialize Qdrant collections.
List all repositories.
Quick test of an MCP server with default configuration. Simplified version that uses auto-detected settings and runs basic functionality tests only.
Search repositories using vector similarity.
Test a single MCP server in an isolated container. Validates the repository exists locally, builds/runs it in an isolated Docker container, connects via MCP protocol, discovers and tests all available tools, and returns comprehensive test results.
Test connection to Qdrant server.
Update configuration settings.
Credentials and secrets exposed as tool parameters: 'api_key' in update_settings and 'env' object in create_repository and test_mcp_server risk leaking secrets into agent logs and prompt history.
No error handling guidance in any tool description. Agents have no hint how to recover from failures (e.g., if test_mcp_server fails, should the agent retry, check Docker status, or ask the user?). Error responses will be opaque.
Destructive/irreversible tools (delete_repository, cleanup_test_containers) lack dry-run or confirmation patterns. An agent can irrevocably delete repositories without a safeguard.
Descriptions are too generic and do not distinguish tools or explain composition. E.g., 'Generate YAML configuration' and 'Generate JSON configuration' are two separate tools with nearly identical purposes, descriptions fail to explain when to use each. Similarly, get_qdrant_status, get_collection_status, and test_qdrant_connection are distinct but described minimally.
Parameter 'test_config' in test_mcp_server and batch_test_mcp_servers is typed as object with properties but lacks a description of the object structure. LLMs cannot infer what fields are valid or required inside test_config.
No pagination parameters (limit, offset, page_size, cursor) on list_repositories or search_repos. If the database contains thousands of repositories, these tools could return massive unbounded result sets, exhausting context windows.
Tool composition issue: create_repository and test_mcp_server are separate. Agents must call create_repository, then separately invoke test_mcp_server. This should be composable, or a combined 'create_and_test_repository' tool should be offered for the common case.
No idempotency hints. Tools like create_repository and initialize_collections do not state whether repeated calls are safe. Agents retrying on ambiguous failures risk duplicate repository entries or double-initialization.
Parameter descriptions lack format constraints. E.g., 'repo_url' in fetch_repository_details is described as 'URL of the Git repository' but does not specify the expected format (https://github.com/user/repo vs git@github.com:user/repo vs file:// paths).