Loading ToolBench
VyMCP has well-structured tool definitions with clear naming (verb_noun pattern) and comprehensive descriptions. All 10 tools have descriptions exceeding 20 characters, and input schemas are present with typed parameters. However, output schemas are not documented, parameter descriptions lack detail on constraints/formats, and error handling guidance is absent. The propose_* tools lack confirmation/dry-run patterns despite destructive risk. Tool composition is sound (single responsibility), but missing pagination for list_instances and no guidance on when to call discovery tools.
List the operations available for a feature, for use with propose_operations. Returns each op's name, arg_count, and description, plus any required top-level body fields (e.g. nat_type, group_name). Args: feature: A feature slug from list_features (e.g. "nat", "static-routes").
Version-aware capability flags for a feature on a specific instance. Args: feature: A feature slug from list_features (e.g. "nat", "firewall/ipv4"). instance_id: An instance_id from list_instances.
Normalized configuration for a feature on a specific instance. Args: feature: A feature slug from list_features (e.g. "nat", "firewall/ipv4"). instance_id: An instance_id from list_instances.
List the VyOS configuration features VyMCP can read. Use the returned ``feature`` slug with get_capabilities and get_config.
List the VyOS instances this token can access. Returns each instance's ``instance_id`` (pass it to other tools), name, site, host, VyOS version, and whether it is active.
Output schemas not documented. Tools return dict[str, Any] with no specification of fields, types, or structure. LLMs cannot plan downstream calls or extract required data.
Destructive/write tools (propose_delete_address_group, propose_operations) lack confirmation or dry-run patterns. No explicit guidance that these are irreversible or require user approval before apply_change.
Parameter descriptions lack constraint details. 'addresses' array in propose_create_address_group states '1-50' but no min/max schema constraints. 'feature' slug format not formally specified (regex/enum).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Propose adding members to an existing IPv4 firewall address-group. Does NOT apply.
Propose creating an IPv4 firewall address-group. Does NOT apply. Returns a plan_id to pass to apply_change after the user approves. Args: instance_id: Target instance (from list_instances). name: Group name. addresses: Host IPs or 'start-end' ranges, 1-50 (a VyOS address-group does not accept CIDR networks — those belong in a network-group). description: Optional description.
Propose deleting an entire IPv4 firewall address-group. Does NOT apply. Deleting a group in use by firewall rules can break those rules — review carefully.
Propose a batch of operations for any feature. Does NOT apply. Each operation is ``{"op": <name>, "value": <str>}`` where op comes from describe_feature_operations. Two-argument ops take a comma-joined value. Returns a plan_id to pass to apply_change after the user approves. Args: feature: A feature slug (e.g. "nat", "static-routes"). instance_id: Target instance from list_instances. operations: The operations to perform. fields: Required top-level body fields for the feature, if any (see describe_feature_operations).
Propose removing members from an IPv4 firewall address-group. Does NOT apply.
No error handling guidance. Tools return dict[str, Any] with no documented error cases, recovery steps, or actionable error messages. LLMs cannot self-correct on invalid feature slugs or instance IDs.
list_instances lacks pagination. No limit, offset, or cursor parameters. Large deployments could return unbounded results, exhausting context window.