Loading ToolBench
Understand any codebase in seconds - MCP server for AI coding agents
CodeGraph MCP provides 7 code analysis tools with clear purposes but exhibits significant gaps in definition quality. All tools have descriptions and JSON schemas present, but descriptions are generic (averaging 110 chars, below the 194-char baseline), parameters lack consistent type safety and validation constraints, and output schemas are not explicitly documented. Tool names follow verb_noun convention ('analyze_architecture', 'get_dependency_graph', 'find_security_issues') which is positive, but descriptions fail to clearly explain WHEN to use each tool vs alternatives, what the agent should do with results, or what dependencies exist between tools. Parameter descriptions are minimal: 'path' is described only as 'Path to the project root' without format expectations or validation rules; 'severity' in find_security_issues has an enum but no description of what each level means operationally. No output schemas are documented, agents cannot know what fields to extract from results. Error handling is not evidenced in the code samples provided. The tool registry architecture (ToolRegistry class) is clean but does not demonstrate idempotency guarantees, permission gates, or audit trails.
Detect the architectural pattern of a codebase (MVC, microservices, clean architecture, etc.)
Calculate cyclomatic complexity and other metrics for files
Detect unused code: imports, functions, files
Scan the codebase for security vulnerabilities and hardcoded secrets
Generate visual diagrams of the codebase structure
Generate a dependency graph showing how files and modules import each other
Answer natural language questions about the codebase
Output schemas not documented. Tools return complex nested structures (e.g., getDependencyGraph returns graph objects, findSecurityIssues returns {issues, summary}) but agents cannot know what fields to extract. This forces LLMs to guess at response structure and risks incorrect field access.
Parameter descriptions are too generic and lack validation constraints. 'path' parameter appears in all 7 tools but is described only as 'Path to the project root' or 'Path to analyze', no mention of whether it accepts relative vs absolute paths, symlinks, or what happens if path doesn't exist. 'severity' enum in find_security_issues has no explanation of what each level filters (critical=CVE-exploitable?, high=authentication-bypass?, etc.).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Descriptions are too brief and lack WHEN/WHY context. 'Detect the architectural pattern of a codebase' (analyze_architecture) does not explain when an LLM should call this vs query_codebase. 'Generate a dependency graph' (get_dependency_graph) does not clarify difference from generate_diagram with type='dependencies'. No descriptions explain what happens if the codebase is too large, if results are cached, or if analysis is deterministic.
No error handling guidance in tool definitions. If a path doesn't exist, if analysis times out, or if the security scanner encounters a syntax error, the code provides no recovery hints. BaseTool.getRequiredArg() throws 'Missing required argument' but this is caught and handled at what layer? Agents see a raw error with no next steps.
Parameter interdependencies undocumented. In generate_diagram, 'type' parameter determines what is returned, and 'format' interpretation may differ by type (e.g. 'flow' type says 'not yet implemented' for some formats). No parameter description mentions these dependencies; LLMs will call invalid combinations.
No idempotency guarantees documented. Are these analysis tools idempotent? If an LLM retries analyze_architecture on the same path, does it return the same result? Or does it re-scan and potentially return different results if files changed? No specification prevents agents from assuming idempotency.
Result limits not enforced or documented. getDependencyGraph allows 'maxDepth' parameter but no 'limit' or 'max_results' cap. If a codebase has 10,000 files, does it return all edges? No tool description states a maximum result size, risking context window exhaustion. Baseline pattern requires stating 'Results capped at 50 items; use pagination.'