Loading ToolBench
Server exposes 21 pwntools utilities with consistent naming (verb_noun pattern: p8, u16, asm, disasm, shellcraft, etc.) and complete parameter schemas. All tools have descriptions and typed inputs. However, descriptions are terse (10-50 chars for most tools), lack context on WHEN to use each tool, and omit error guidance. Output schemas are not documented, LLMs cannot predict return structure. No parameter validation constraints (enums, ranges, patterns) despite many parameters accepting only specific values (e.g., 'endian' should be enum ['little', 'big']). Error handling is absent, no recovery guidance if asm() fails or shellcraft_type is invalid. Tools like 'fit' accept 'object' type with no schema detail, forcing LLMs to guess structure. Security: context_set modifies global state, risking cross-call interference in concurrent scenarios.
Assemble code to bytes.
Set pwntools context.
Generate cyclic pattern.
Find offset in cyclic pattern.
Disassemble bytes to code.
Encode bytes to hex string.
Fit data into buffer.
Flatten arguments into bytes.
Descriptions are uniformly terse (10-50 chars). Rubric baseline for A+ tools is 50-200 chars with context on WHEN to use. Current descriptions lack guidance on tool selection, prerequisites, and error recovery.
Parameter 'endian' appears in 8 tools but is not declared as enum ['little', 'big']. Free-form strings invite hallucinated values. Rubric requires enums for known-set parameters.
Output schemas are not documented. Tools return strings (hex, assembly, shellcode) or integers, but LLMs cannot predict structure or plan downstream calls. Rubric requires documented return types for all tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Hexdump utility.
Pack an integer into 2 bytes.
Pack an integer into 4 bytes.
Pack an integer into 8 bytes.
Pack an integer into 1 byte.
Generate shellcode.
Generate dup shellcode with execve.
Generate setreuid shellcode.
Unpack 2 bytes to integer.
Unpack 4 bytes to integer.
Unpack 8 bytes to integer.
Unpack 1 byte to integer.
Decode hex string to bytes.
Parameter 'fit' accepts type 'object' with no schema detail. Rubric requires structured objects with typed fields. LLMs cannot infer the expected dictionary structure (offset→value mapping).
No error handling or recovery guidance. If asm() fails (invalid syntax), shellcraft_type is unrecognized, or cyclic_find() pattern not found, tools return raw exceptions. Rubric requires actionable error messages and recovery hints.