Loading ToolBench
MCP server toolkit for AI agents - surgical code analysis and modification
Code Scalpel presents 22 tools with schemas present but significant quality gaps across naming, descriptions, and output documentation. Tools are named with reasonable verb-noun patterns (analyze_, get_, scan_, extract_), but descriptions are generic and brevity-violating. Most critically: NO OUTPUT SCHEMAS ARE DOCUMENTED. The rubric requires documented return types for all tools (100% of A+ tools have them); without visible output schemas, LLMs cannot reason about chaining tools or extracting fields. Input schemas are present but parameter descriptions are minimal (typically 5-15 words), falling short of the 72-char average baseline. No tool has error handling guidance, recovery paths, or actionable error messages. The 22 tools cluster into overlapping concerns (multiple graph/analysis tools with unclear distinctions), violating single-responsibility and composition principles. Pagination, limits, and result caps are undocumented. Security-sensitive tools (rename_symbol, update_symbol) lack dry-run details or confirmation flow. Overall, this server reads as a thin wrapper around internal APIs with minimal LLM-optimization.
Analyze code structure and properties using AST/tree-sitter parsing
Check code against organization policies
Crawl project files and build index
Scan for security issues across multiple files (Pro/Enterprise tier)
Extract specific code elements from files or across files
Generate unit tests for given code
Generate call graph for functions and methods
NO OUTPUT SCHEMAS DOCUMENTED. Rubric baseline: 100% of A+ tools have documented return types. This server defines input schemas but provides zero visibility into what each tool returns. LLMs cannot chain tools (e.g., extract_code → rename_symbol) without knowing what fields are available. This is a critical blockers for agent reasoning.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 53 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 40 | 1.23.0 or higher (mcp>=1.23.0 specified in dependencies)+ | v1 |
Get cross-file dependencies and imports
Get file context and metadata
Get neighborhood of a node in program dependence graph
Get project structure and file organization map
Find all references to a symbol in the project
Rename a symbol across the project (Pro/Enterprise tier)
Scan project dependencies for vulnerabilities
Scan code for security vulnerabilities and issues
Simulate and analyze proposed code refactorings
Perform symbolic execution on code to explore execution paths
Detect type annotation loss and type safety issues
Detect unified sinks and taint propagation in code
Update symbol definition or usage (Pro/Enterprise tier)
Validate file paths and resolve them relative to project root
Verify code policy configuration integrity
GENERIC, MINIMAL DESCRIPTIONS. Example: 'Detect unified sinks and taint propagation in code' (10 words, ~60 chars). Rubric baseline: avg 194 chars, p10=34, p90=392. Most descriptions under 20 words; none explain WHEN to use the tool vs. alternatives (e.g., when to call security_scan vs. unified_sink_detect vs. cross_file_security_scan), WHAT the tool returns, or what happens on failure. LLMs cannot distinguish overlapping tools without richer context.
PARAMETER DESCRIPTIONS MISSING OR TRIVIAL. Example: extract_code's 'target_type' has description 'Element type to extract (function, class, method, etc.)', barely distinguishes from the parameter name. Rubric baseline: 94% of A+ tools have descriptions on all params; average param annotation ~72 chars. Most params here are 5-15 word fragments. For example, 'code' → 'Source code to analyze' does not specify encoding, max size, or what happens if code is malformed.
MISSING ENUM CONSTRAINTS & VALIDATION GUIDANCE. 'language' parameter in analyze_code, security_scan, etc. lists 13 supported languages in the description as prose ('python, javascript, java, ...'). Rubric pattern: enums must be declared formally, not listed as examples. This invites hallucinated values (e.g., 'perl', 'scala', 'rust' when rust is listed but not supported). No per-param error messages specified (e.g., 'Invalid language: got X, must be one of Y').
NO ERROR HANDLING OR RECOVERY GUIDANCE. No tool documents error cases, error codes, or what the LLM should do on failure. Example: extract_code could fail if file_path is invalid, target_name is not found, or cross-file resolution times out, but no guidance on recovery (e.g., 'Try get_project_map() to validate file paths'). Rubric pattern: 'Error responses must tell the LLM what to do next.' This server provides none.
DESTRUCTIVE TOOLS LACK DRY-RUN & CONFIRMATION FLOW. rename_symbol and update_symbol carry WRITE risk but descriptions do not explain confirmation behavior or rollback mechanics. Rubric pattern: 'Irreversible operations should support a dry-run or confirmation step.' Both tools have a 'dry_run' parameter, but there is no documentation of what dry-run returns, whether it actually modifies state, or how the LLM confirms before executing. No mention of rollback/undo.
PAGINATION & RESULT LIMITS UNDOCUMENTED. Tools like crawl_project, get_project_map, get_symbol_references likely return large result sets, but no limit, offset, cursor, or page_size parameters are mentioned. Rubric pattern: 'Tools returning lists should accept page/offset and limit parameters and return a total count or next_cursor.' Returning unbounded lists risks context window exhaustion and LLM hallucination.
OVERLAPPING & AMBIGUOUS TOOL BOUNDARIES. At least 5 security-scanning tools (security_scan, unified_sink_detect, cross_file_security_scan, type_evaporation_scan, code_policy_check) have overlapping purpose. No descriptions clarify when to use one vs. another. Rubric principle: 'When multiple tools operate on the same resource, their names must make the distinction obvious.' Example: is type_evaporation_scan a subset of security_scan, or does it check different invariants? Unclear.
NO TOOL COMPOSITION OR CHAINING GUIDANCE. Descriptions do not hint at tool dependencies. Example: extract_code requires 'file_path', but no description says 'If you don't know the file path, call get_project_map() first.' Rubric pattern: 'Include dependency hints to prevent wasted calls and guide multi-step planning.' This server is silent on prerequisites and natural call sequences.
INSUFFICIENT PARAMETER CONSTRAINTS. 'max_depth' in get_project_map is an integer with no min/max specified. Unbounded integers let LLMs pass absurd values (e.g., max_depth=999999) that break timeouts or memory limits. Rubric rule: 'Specify minimum and maximum for numeric parameters.' Example: 'max_depth: integer, range 1 - 10, default 3' with rationale.