Loading ToolBench
The VyOS MCP server demonstrates solid definition quality with comprehensive tool coverage (19 tools), well-structured input models using Pydantic, and consistent parameter descriptions. All tools have descriptions ranging from 50-200+ characters and explicit input schemas with type definitions. Tool annotations are present (readOnlyHint, destructiveHint, idempotentHint). However, several issues limit the score: (1) tool names are overly prefixed with 'vyos_' which is redundant and reduces clarity (all tools in this domain are VyOS operations); (2) some descriptions could be more concise and action-focused (current average ~150 chars, baseline optimal 50-200 chars); (3) output schemas are not documented in tool definitions, responses are JSON strings without explicit schema declarations; (4) error handling is centralized but lacks actionable recovery guidance in some tools; (5) composition could be improved, vyos_image_add and vyos_image_delete are split but share the same input model with conditional logic (ImageInput with 'op' enum), suggesting a combined tool might be clearer; (6) vyos_reboot and vyos_poweroff have identical input structure and could potentially be unified. Per-tool analysis shows strong consistency: 17/19 tools have complete schemas and descriptions, but 2 tools (vyos_reboot, vyos_poweroff) exhibit schema redundancy.
Apply multiple configuration operations in a single atomic commit. Attempt up to min(5, len(operations)) edits to repair any user errors, retrying the entire batch if changes are made.
Add or modify a comment on a configuration node and commit. Args: path: config path segments confirm_time: optional commit-confirm timeout in minutes
Confirm a commit-confirm pending commit. Use after vyos_set_config, vyos_delete_config, or vyos_batch_configure with a confirm_time parameter to permanently apply the changes. If not called before the timeout, changes auto-revert.
Delete a single configuration node and commit. Args: path: config path segments to delete confirm_time: optional commit-confirm timeout in minutes
Check whether a configuration path exists. Returns true/false. Useful before setting or deleting configuration.
Run a generate command (e.g. SSH keys, certificates, etc.). Examples: - Generate SSH keys: path=["system", "ssh", "client-key", "rsa"] - Generate WireGuard keys: path=["interfaces", "wireguard", "wg0", "private-key"]
Tool names are unnecessarily prefixed with 'vyos_' (all 19 tools). This is redundant since the server domain is VyOS management. Shorter names like 'show_config', 'set_config', 'batch_configure' would be clearer and reduce cognitive load when LLMs parse tool lists.
Output schemas are not documented. While input schemas are comprehensive (Pydantic models with types and descriptions), response structures are returned as JSON strings via _json() helper without formal schema declarations. LLMs cannot infer the structure of responses (e.g., vyos_show_config returns JSON but the shape is not declared). This violates the 'document the output schema' rule from pattern:tool.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 72 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
Download and install a system image from a URL. Args: url: HTTP(S) URL to the image file
Delete a system image by name. Args: name: image name to remove
Get public system information (version, hostname, banner). No authentication required. Useful to verify connectivity.
Load configuration from a file on the VyOS system. This replaces the running config. Use with caution.
Merge configuration from a file or inline string. Combines the source config with the running config (no replacement).
Power off the VyOS system (destructive operation). The router will be powered down. Configuration is saved before poweroff.
Reboot the VyOS system (destructive operation). The router will go offline. Configuration is saved before reboot.
Run a reset command. Examples: - Reset BGP session: path=["bgp", "1000"] - Reset OSPF: path=["ospf"]
Return values of a multi-valued configuration node. Use this for nodes that hold multiple values (e.g. interface addresses, DNS servers, NTP servers).
Save the running configuration to disk. Args: file: optional file path to save to (default /config/config.boot)
Set a single configuration value and commit. Args: path: config path segments, e.g. ["interfaces", "ethernet", "eth0", "address", "10.0.0.1/24"] confirm_time: optional commit-confirm timeout in minutes
Run an operational-mode show command. Examples: - Show interfaces: path=["interfaces"] - Show running config: path=["configuration"] - Show routing table: path=["route"] - Show BGP summary: path=["bgp", "summary"] - Show interfaces brief: path=["interfaces", "brief"]
Retrieve the running VyOS configuration (full or partial). Pass an empty path list for the entire config tree, or specify path segments to retrieve a subtree.
vyos_image_add and vyos_image_delete share an ImageInput model with an 'op' enum ('add'/'delete') and conditional parameters (url required for 'add', name required for 'delete'). This couples two distinct operations. Per pattern:tool, each tool should do exactly one thing. The conditional logic and field validator at MergeConfigInput.file_or_string_required shows awareness of parameter relationships, but using an enum discriminator across tool boundaries is poor composition. Recommend: two separate tools without the 'op' enum.
vyos_reboot and vyos_poweroff have identical input schemas (action enum + confirm boolean) with only the enum value differing. This is borderline redundant and could confuse LLMs. Both require explicit confirmation (confirm=true), which is good for safety, but the parallel structure suggests a unified 'system_control' tool with action='reboot'|'poweroff' might be clearer. Alternatively, separate tools should have non-overlapping parameter names to signal distinctness.
Error handling is centralized (_format_error) but lacks actionable recovery guidance in tool descriptions. When a tool fails (e.g., vyos_set_config with invalid path), the error message may not guide the LLM on next steps. Example: if a path does not exist, should the LLM call vyos_exists() first? This is not documented in tool descriptions. Per pattern:recovery-guide, errors should include suggested remediation steps.
vyos_batch_configure description states 'Attempt up to min(5, len(operations)) edits to repair any user errors, retrying the entire batch if changes are made.' This auto-repair behavior is not standard and could cause unexpected side effects if the LLM's intent is modified during retry. The description does not explain what 'edits to repair' means or which errors are retryable. This violates the clarity requirement from pattern:tool-description.
MergeConfigInput has a field_validator that requires 'file' OR 'string' but the error message 'Provide either file or string (or both)' is not exposed to the LLM in tool description. LLMs will see 'Inline VyOS config string to merge' and 'File path on VyOS to merge from' but may not understand that at least one is required. Add explicit mutual-requirement language to both parameter descriptions.