Loading ToolBench
MCP server for IDA Pro reverse engineering tool providing disassembly, decompilation, cross-references, search, modification, memory, types, stack frames, and debugging capabilities
IDA-MCP demonstrates weak definition quality across most tools. While 6 tools are registered with basic descriptions and parameter annotations via Pydantic Field(), the schemas lack rigor, descriptions are minimal, and parameter semantics are poorly documented. Tool names are clear (verb_noun pattern holds), but descriptions average ~120 chars (below the 194 baseline for production tools) and often omit actionable context for LLM decision-making. Parameters lack enum constraints where applicable (e.g., 'port' could default/validate better), and output schemas are implicit rather than documented. Error handling is absent, no guidance on recovery, retryability, or what conditions cause failures. The server architecture exposes implementation details (port/timeout parameters forwarded to all backend tools) that should be abstracted. Most critically, the proxy forwarding pattern (register_tools.py) creates tools without complete visibility into their actual schemas, capping them at 50 per the rubric.
Health check. Returns {ok: bool, count: int} where count is number of registered IDA instances.
Close the target IDA instance. Warning: This terminates the process.
List all registered IDA instances. Returns array of {id, port, pid, input_file, started, ...}.
Launch IDA Pro with the specified file. Automatically attempts to load IDA-MCP plugin.
Choose the default IDA instance port for subsequent calls. If port omitted, auto-selects (prefer 10000). Returns {selected_port} or {error}.
Request shutdown of the standalone gateway. Refuses while instances are registered unless force=true.
Output schemas are inferred from descriptions, not formally declared. No tool exposes a structured JSON Schema for its return type in the code.
Descriptions are below production baseline (avg 194 chars). Most descriptions range 95 - 142 chars and lack LLM decision context: when to call this vs alternatives, what conditions cause errors, what the output fields mean.
No error handling guidance. Tools lack descriptions of failure modes (e.g., 'file not found', 'port already in use', 'permission denied') and no recovery hints (e.g., 'Try list_instances() to find available ports').
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-03-09 | C | 60 | - | v1 |
Proxy forwarding pattern exposes generic 'port' and 'timeout' parameters on all backend-forwarded tools without justification. These are implementation details that clutter the schema. Backend tools should not require port/timeout parameters if a default is already selected.
Parameter descriptions lack validation constraints. Numeric parameters (port, timeout) don't specify valid ranges. File path parameters don't state accepted formats or existence expectations.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present, even though close_ida and shutdown_gateway are clearly destructive operations. LLMs cannot infer risk without explicit hints.
Inferred tool schemas (via register_tools.py forwarding mechanism) are not directly visible in source. Tool definitions are dynamically constructed from ToolSpec without complete schema visibility in registration.