Loading ToolBench
MCP server that enables LLM-assisted reverse engineering through code execution against binaries open in Binary Ninja
The 'execute' tool has an exceptionally detailed and comprehensive description, well above the 194-char baseline (1800+ chars). However, the tool conflates multiple distinct concerns into a single interface: binary analysis, code execution, workspace management, and skill management. The input schema is minimal (only 'code' and optional 'description'), which masks significant complexity. Parameter descriptions exist but the schema itself lacks the structured output documentation that would guide LLM planning. The tool's destructive capability (patching binaries, modifying state) is flagged but lacks explicit dry-run or confirmation patterns. The tool name 'execute' is generic and does not clearly convey that it runs arbitrary Python against a binary analysis tool.
CODE MODE: execute Python against the binary open in Binary Ninja. Executes Python code with access to Binary Ninja's API through several namespaces: binja for analyzing the binary, workspace for files that outlive a call, skills for saved code, bv for the raw BinaryView and bn for the binaryninja module.
Tool name 'execute' is generic and does not follow verb_noun convention. Does not clearly indicate what is being executed or against what context. 'analyze_binary_with_python' or 'run_binja_analysis' would be more actionable.
Multiple distinct concerns conflated into a single tool: executing arbitrary code, managing workspace files, saving/loading skills, managing checkpoints, and modifying binaries. This violates the single-responsibility principle. Should be split into at least: execute_analysis, manage_workspace, manage_skills, manage_checkpoints.
Output schema is not documented. Tool returns 'print() output and last expression value, truncated to ~6000 tokens' but LLMs have no structured way to know what fields or types to expect. This forces the LLM to parse unstructured output.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 54 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 52 | 2024-11-05+ | v1 |
Tool performs irreversible operations (binary patching, file writes, checkpoint rollback) but lacks a dry-run mode or confirmation step. Agents can destroy analysis work without recovery. No atomic transaction pattern documented.
Input parameter 'description' is optional and its purpose is unclear. The description says 'What this code does' but it is not used in execution and appears to be metadata only. If metadata, should be separated from execution intent.
No error handling guidance. Tool can fail for many reasons (syntax error in code, API misuse, missing functions, binary not loaded) but does not explain to the LLM what the failure means or how to recover. No error categorization (retryable vs fatal).
Tool accepts arbitrary Python code with access to the binaryninja module and file system. No input validation or sanitization documented. An LLM could be tricked into executing malicious code (e.g., 'import os; os.system(...)') or exfiltrating the binary.
The 'code' parameter lacks format/constraint documentation. Description says 'Python code' but does not explain: valid syntax, forbidden imports, execution timeout, memory limits, or what happens if code takes too long.