Loading ToolBench
This server exposes 20 tools wrapping python-nmap's PortScanner API. While all tools have descriptions and basic schemas, the quality is significantly below production baseline. Tool names lack clear verb-noun structure (e.g., 'port_scanner_scan' vs 'scan_ports'). Descriptions are present but generic and often under 100 chars, providing minimal context for LLM selection. Parameter descriptions exist but are sparse and lack constraints (no enums, ranges, or format guidance). Output schemas are not documented, responses are returned as dicts with inconsistent structure (some include 'success' flags, others don't). Error handling is minimal: most tools return {'success': False, 'error': str(e)} without recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear read/write semantics. The server conflates three separate scanner implementations (sync, async, yield) into one tool namespace, creating cognitive load for LLM selection.
Return a sorted list of all hosts that have been scanned.
Analyse NMAP xml scan output.
Initialize the PortScannerAsync instance.
Scan given hosts in a separate process and return host by host result using callback.
Check if async scan is still running.
Stop async scan.
Wait for async scan to complete.
Tool names lack clear verb-noun structure. Names like 'port_scanner_scan' and 'port_scanner_get_item' bury the action verb and repeat 'port_scanner' prefix, reducing clarity. LLMs struggle to infer intent from names alone.
Output schemas are not documented. Tools return dicts with inconsistent structure (some include 'success' flag, others return raw data). LLMs cannot plan downstream calls or extract required fields reliably.
Parameter descriptions lack constraints. No enums for 'arguments', no ranges for 'timeout', no format guidance for 'hosts' or 'ports'. LLMs cannot validate input and may pass invalid nmap syntax.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 40 | 2026-07-28+ | v2 |
Return the command line used for the scan.
Return CSV output as text.
Get host scan details.
Check if host has scan results.
Initialize the PortScanner instance.
Return the last text output of nmap in raw text.
Do not scan but interpret target hosts and return a list of hosts.
Return nmap version if detected (int version, int subversion) or (0, 0) if unknown.
Scan given hosts using nmap.
Return scaninfo structure.
Return scanstats structure.
Initialize the PortScannerYield instance.
Scan given hosts and yield results progressively.
Error handling provides no recovery guidance. Errors return {'success': False, 'error': str(e)} without actionable next steps. LLMs cannot determine if error is retryable, user-fixable, or fatal.
Three scanner implementations (sync, async, yield) are exposed as separate tool sets in one namespace, creating cognitive load. LLMs must reason about which variant to use without clear guidance on trade-offs.