Loading ToolBench
VMware NSX networking management with 33 MCP tools: segments, gateways, NAT, routing, IPAM.
The VMware NSX MCP server exhibits significant quality gaps across definition and specification. While it provides 35 tools covering a broad NSX management domain (segments, gateways, routing, NAT, IPAM), critical issues undermine usability: (1) Tool definitions are inferred from test/eval files rather than directly visible in the server registration code; (2) Descriptions vary wildly in quality, some are exceptional (e.g., delete_static_route with detailed blast_radius guidance), others are one-liners with no actionable context; (3) Input schemas are inconsistent, some tools (create_static_route, delete_static_route) show detailed typed parameters with enum constraints, but many read-only tools have no visible schema specifications; (4) No documented output schemas for any tool, forcing LLMs to guess what fields are returned; (5) Parameter naming has inconsistencies (tier1_id vs gateway_id for the same concept across different versions of the same tool); (6) Error handling guidance is present in a few tools but absent in most; (7) The codebase shows tool definitions scattered across tests/eval/capability/_family.py and various tool modules, making it unclear which definition is canonical. Per-tool analysis reveals an average quality score of 42, most tools cluster in the 35 - 55 range, with exceptional outliers (delete_static_route reaches ~65) balanced by bare-bones definitions (many list_* tools at ~25 - 30). This is typical of early-stage community MCP servers that prioritize feature breadth over definition depth.
[WRITE] Configure BGP on a Tier-0 gateway.
Create an IP pool via Policy API (PUT).
Create a NAT rule on a Tier-1 gateway via Policy API (PUT).
[WRITE] Create a logical segment.
Create a static route on a gateway via Policy API (PUT).
[WRITE] Create a static route on a Tier-0 or Tier-1 gateway via the Policy API. Use this for destinations not covered by connected or advertised routes, e.g. a VPN or external subnet. Run list_static_routes first to avoid an id clash — the same route_id overwrites (PUT). For the Tier-0 to advertise a Tier-1's static route upstream, the gateway needs TIER1_STATIC_ROUTES advertisement, set via update_tier1_gateway. Returns the created route dict, else {"error", "hint"}. Then confirm with list_static_routes; delete_static_route is the inverse.
Tool definitions inferred from test/eval files rather than directly visible in server registration. The server.py file does not show explicit @mcp.tool() decorators with inline parameter schemas; instead, tools are imported from vmware_nsx.mcp_server.tools submodules. Test definitions in tests/eval/capability/_family.py are used as reference, but these are not the canonical runtime definitions. This violates the principle of self-documenting code and makes it impossible to verify that production tool schemas match documentation.
17 of 25 read-only tools (list_segments, get_segment, list_tier0_gateways, list_tier1_gateways, list_transport_zones, list_transport_nodes, list_edge_clusters, list_nat_rules, get_bgp_neighbors, list_static_routes, list_ip_pools, get_ip_pool_usage, list_nsx_alarms, get_transport_node_status, get_edge_cluster_status, get_nsx_manager_status, get_logical_port_status, get_segment_port_for_vm) have only one-line descriptions ('Read-only tool: ...') with no actionable guidance on WHEN to call them, WHAT they return, or HOW to use results downstream. This forces LLMs to guess the structure and purpose.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 39 | <=2025-11-25 | v2 |
[WRITE] Create a Tier-1 gateway.
[WRITE] Delete an IP pool.
Delete a NAT rule from a Tier-1 gateway.
[WRITE] Delete a logical segment.
[WRITE] Permanently delete a static route from a Tier-0 or Tier-1 gateway. Irreversible: traffic to the route's destination CIDR immediately falls back to remaining routes or is dropped. Without confirm=True this only previews: it returns blast_radius (the gateway, the route's network and next hops, blockers, unmeasured) and deletes nothing. Show that to the user and get their decision. Do not set confirm=True on your own because the user asked to delete earlier: they have not seen the blast radius yet. gateway_type must match where the route lives; a route_id not on that gateway is an error, and confirm=True refuses when the routes could not be read. Returns {"action": "preview" | "deleted", "blast_radius": ...}, else {"error", "hint", "blast_radius"?}.
Delete a static route from a gateway.
[WRITE] Delete a Tier-1 gateway.
Read-only tool: get BGP neighbor status.
Read-only tool: get edge cluster status.
Read-only tool: get IP pool usage statistics.
Read-only tool: get logical port status.
Read-only tool: get NSX Manager status.
Read-only tool: get details of a specific segment.
Read-only tool: get logical port for a VM.
Read-only tool: get details of a specific Tier-0 gateway.
Read-only tool: get details of a specific Tier-1 gateway.
Read-only tool: get transport node status.
Read-only tool: list edge clusters.
Read-only tool: list IP pools.
Read-only tool: list NAT rules on a gateway.
Read-only tool: list NSX alarms.
Read-only tool: list NSX logical segments.
Read-only tool: list static routes on a gateway.
Read-only tool: list Tier-0 gateways.
Read-only tool: list Tier-1 gateways.
Read-only tool: list transport nodes.
Read-only tool: list transport zones.
[WRITE] Update a logical segment.
[WRITE] Update a Tier-1 gateway.
No output schemas documented for ANY tool. The input parameter schemas are present for some write/create tools (create_static_route, create_nat_rule, create_ip_pool), but there is no corresponding documentation of what fields are returned. LLMs cannot plan downstream tool calls or extract necessary chaining IDs without knowing output structure. E.g., does create_segment return {id, name, status} or {uuid, display_name, realization_state}?
Duplicate and inconsistent tool definitions. Tools like create_static_route and delete_static_route appear twice with different parameter schemas: one version uses (tier1_id, route_id, network, next_hop, gateway_type, target), the other uses (gateway_id, route_id, network, next_hops: array). The parameter naming and structure are inconsistent, which will confuse LLMs about which parameters to pass.
Parameter naming inconsistency across similar tools. Some tools use 'tier1_id', others use 'gateway_id' for the same concept. Some accept 'next_hop' (singular, string), others accept 'next_hops' (plural, array). This forces LLMs to reason about parameter mappings and invites misuse.
Write/destructive tools lack comprehensive error guidance. Most write tools (create_segment, update_segment, delete_segment, create_tier1_gateway, etc.) have minimal descriptions and no documented recovery paths. E.g., delete_segment offers no guidance on what happens if the segment is in use, what 'blast_radius' means, or how to undo the operation.
Tool annotations present (toolAnnotations=true) but not visible in schema definitions. The config claims tool annotations are enabled, but the provided parameter schemas do not show readOnlyHint, destructiveHint, or idempotentHint markers. This suggests annotations may be defined elsewhere in the tool modules, but their absence in the visible specs means they are not verifiable.
No pagination guidance for list_* tools. Tools like list_segments, list_tier0_gateways, list_tier1_gateways, list_nat_rules, list_static_routes, list_ip_pools, list_nsx_alarms have no documented limit, offset, page_size, or cursor parameters. If NSX environments contain hundreds of these objects, LLMs will be flooded with unmanageable response sizes.
Parameter descriptions for write tools are sparse. create_nat_rule, delete_nat_rule, create_ip_pool, and others have short or missing parameter descriptions. E.g., create_nat_rule's 'source_network' says 'Source CIDR for matching (used in SNAT/NO_SNAT)' but does not clarify WHEN this parameter is required or what format is valid. This violates the rubric guideline that every parameter needs a description explaining what it controls.