Loading ToolBench
MCP server that exposes guarded Cisco pyATS operations to AI assistants such as GitHub Copilot.
Server has 5 well-named tools with clear descriptions and documented input schemas. Tool names follow verb_noun pattern (list_devices, run_show_command, health_check, backup_running_config, apply_config). All tools have descriptions 50-200 chars, meeting LLM-optimization baseline. Input parameters are typed and described. However, output schemas are not documented, LLMs cannot predict response structure. Error handling is minimal; no recovery guidance or actionable error messages. Missing tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles. Parameter constraints lack enums where applicable (e.g., 'level' in health_check accepts 'basic'|'detailed' but no enum declared).
Apply configuration lines to a device. Disabled unless explicitly enabled. Requires `PYATS_MCP_ALLOW_CONFIG=true`. Saving to startup-config additionally requires `PYATS_MCP_ALLOW_SAVE=true`. Service-impacting commands such as `reload`, `write erase` and local user changes are always rejected.
Return the running configuration of a device for offline backup or diffing.
Collect a read-only health snapshot from a device.
List the devices in the pyATS testbed that this server may reach.
Run a single read-only `show` command on a device and return its output.
Output schemas not documented. LLMs cannot predict response structure or plan downstream tool calls. All 5 tools return strings with no documented field structure.
Missing tool annotations. apply_config is destructive (WRITE risk) and health_check/run_show_command are read-only, but no readOnlyHint or destructiveHint declared in schema. Agents cannot infer safety profile.
Parameter 'level' in health_check accepts free-form string with no enum constraint. Description says 'basic' or 'detailed' but no formal enum declared. LLMs may hallucinate invalid values.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | <=2025-11-25 | v2 |
Error handling lacks recovery guidance. CommandNotAllowedError and DeviceNotAllowedError are raised but no actionable messages guide LLM on next steps (e.g., 'Set PYATS_MCP_ALLOW_CONFIG=true to enable config changes').
No pagination support. run_show_command, health_check, and backup_running_config return truncated output (max_output_chars) but no mechanism to retrieve remaining data. Large configs are silently truncated.