Loading ToolBench
Model Context Protocol server for Binary Ninja - enables seamless integration of Binary Ninja's capabilities with MCP clients
Binary Ninja MCP has clear naming conventions (verb_noun style: list_methods, get_entry_points, search_functions_by_name, decompile_function, etc.) and explicit tool descriptions. However, there are significant gaps in parameter documentation, output schema clarity, and error handling guidance. 10 of 11 tools have descriptions ranging from 65-150 characters, meeting baseline minimums. All tools use Zod schemas with type definitions and parameter descriptions. The schema quality is moderate, parameters have types and brief descriptions, but output schemas are not formally documented or returned in structured format. Error handling exists but is minimal, returning generic text responses without recovery guidance. Most tools return unstructured text via { type: 'text', text: '...' } rather than structured JSON, limiting LLM's ability to extract and chain data.
Decompile a specific function by name and return the decompiled C code.
Retrieve the disassembled code of a function as assembly mnemonic instructions.
List entry point(s) of the loaded binary.
Get IL for a function in the selected view (hlil, mlil, llil).
List all function names in the program with pagination.
Rename a data label at the specified address.
Rename a function by its current name. The configured prefix will be automatically prepended if not present.
rename_multi_variables has confusing parameter schema with three overlapping input options (mapping_json, pairs, renames_json). LLMs cannot determine which to use; schema lacks enum or mutual exclusivity constraint. Violates single-responsibility principle.
All tools return unstructured text responses (type: 'text') instead of structured JSON objects. LLMs cannot reliably extract function addresses, counts, or hierarchical data. Violates response-shaper pattern, limits chaining capability.
Output schemas are not documented. LLMs have no way to know what fields list_methods returns, what structure get_entry_points provides, or what rename_function confirms. Pattern baseline requires documented return types for 100% of A+ tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 46 | - | v1 |
Rename multiple local variables in one call.
Rename a variable in a function.
Search for functions whose name contains the given substring.
Set a comment at a specific address.
Error handling is generic and non-actionable. Errors return plain text like 'Error: no response' or 'Error: {error}'. LLMs receive no guidance on whether to retry, call a different tool, or ask the user. Violates recovery-guide pattern.
No pagination documentation for list_methods. While offset/limit params exist, tool description does not mention pagination API contract, whether results are sorted, or what total count is returned. Baseline requires pagination explanation in description.
get_il accepts name_or_address as a single string parameter. Tool must detect 0x prefix or numeric format to disambiguate internally, but description does not explain this logic clearly. Parameter documentation should explicitly state acceptance of both forms.
No dry-run or confirmation step for destructive operations (rename_function, rename_single_variable, rename_multi_variables, rename_data, set_comment). Agents can accidentally rename critical functions or overwrite metadata without confirmation. Violates confirmation-request pattern.
Tool descriptions do not clarify state modification implications. rename_* tools should explicitly state 'This modifies the binary database and cannot be undone' or similar. Users/LLMs need to know which calls are reversible. Baseline: descriptions must say so for command-tools.