Loading ToolBench
Isolated Windows vulnerability-research platform for Kali and AI agents
Winbox exposes 8 tools with destructive/write operations on Windows VMs. Critical gaps: (1) Most tools lack input schemas entirely, setup, provision, applocker enable/disable/status, autologin enable/disable/status have empty or missing parameter definitions in visible code. (2) Descriptions exist but are domain-specific and lack LLM-optimized guidance on WHEN to call each tool vs alternatives. (3) No output schemas documented. (4) No error handling guidance, tools will fail silently or with raw exceptions. (5) No idempotency or confirmation patterns for destructive ops (setup, applocker disable). (6) Tool names lack clear verb-noun structure (e.g., 'applocker enable' should be 'enable_applocker'). Per-tool analysis: setup (schema=0, description=40), provision (schema=0, description=20), applocker_enable (schema=0, description=60), applocker_disable (schema=0, description=60), applocker_status (schema=0, description=50), autologin_enable (schema=0, description=50), autologin_disable (schema=0, description=40), autologin_status (schema=0, description=50). Average: 32.
Disable AppLocker — clear policy, stop stack, reboot. Reboot is required because appid.sys caches rules in kernel memory and continues enforcing even after AppIDSvc is stopped.
Enable AppLocker with default rules (Exe, Script, MSI, Appx). Standard corporate/exam config: allow C:\Windows\* and C:\Program Files\* for Everyone, Administrators can run anything. No DLL rules. Undo with: winbox applocker disable
Show current AppLocker enforcement status.
Disable auto-login — clears DefaultPassword and ForceAutoLogon.
Enable persistent auto-login for the configured VM user. Writes Winlogon credentials plus the Server 2022 PasswordLess gate. Undo with: winbox autologin disable
Show current autologin status.
Re-run the provisioning script inside the VM.
No input schemas visible for any tool. setup has 4 parameters (windows_iso, yes, os_key, desktop) defined in Click decorators but not exposed as JSON Schema in MCP tool registration. All 8 tools lack formal schema definitions.
Tool names use space-separated format ('applocker enable', 'autologin disable') instead of snake_case verb_noun ('enable_applocker', 'disable_autologin'). LLMs parse verb_noun naming more reliably for intent inference.
No output schemas documented. Tools return unstructured results (status checks return text, setup returns success/failure). LLMs cannot plan downstream calls or extract structured data.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 34 | 2026-07-28+ | v2 |
Build the Windows VM (one-time setup).
Destructive operations (setup, applocker disable, autologin disable) lack confirmation or dry-run patterns. No guidance on recovery if agent makes mistakes. Agents cannot safely retry or undo.
Descriptions lack LLM-optimized guidance on WHEN to call each tool. 'Enable AppLocker with default rules' does not explain when to call enable vs disable vs status, or what happens if already enabled. No dependency hints or prerequisites.