Loading ToolBench
Multi-agent code audit system using LangGraph that analyzes Python codebases for structure, complexity, security, and style issues. Exposes three MCP servers (file_system, code_analysis, git_tools) that work together to scan, analyze, and report on codebase health.
Significant gaps across naming, descriptions, and schemas. Tool names lack clear action verbs (git_summary, run_complexity, run_security, run_lint are vague). Descriptions exist but are verbose (200-300+ chars) and lack structured guidance on when/why to use each tool. Input schemas present but minimal, only 'path' parameter with basic string type, no enums, ranges, or validation hints. Output schemas completely undocumented; LLMs cannot predict what fields to extract. Error handling is absent, subprocess failures return raw stderr or timeout messages without recovery guidance. No security validation of path parameters (path traversal risk). Composition is weak: tools do not chain well (e.g., most_changed_files returns file names but no context for downstream analysis).
Analyze the whole git repo and return a plain-text summary: total commit count, the last commit date (relative, e.g. "5 days ago"), and the list of contributors. Use this to judge how active or abandoned a project is.
List files in a directory (inferred from usage in scanner agent)
Return the files changed most often in the repo's history (hotspots). Files that change frequently tend to be the most bug-prone, so these are worth reviewing closely. Returns the top 10 as "<count> <file>" lines.
Read the contents of a file (used by analyst and reporter agents)
Return the repo's 10 most recent commits, one per line as "<short-hash> <commit message>". Use this to see what work has happened recently in the project.
Analyze cyclomatic complexity of Python code using radon. Returns JSON listing each function/method with its complexity score. Higher scores mean more branches and harder-to-maintain code (roughly: 1-5 simple, 6-10 moderate, 11+ complex, 20+ very risky). Use this to find overly complex functions that should be simplified.
Tool names lack action verbs. 'git_summary', 'run_complexity', 'run_security', 'run_lint' are vague. LLMs cannot infer intent from names alone. Should be 'analyze_git_repo', 'measure_complexity', 'scan_security', 'check_lint'.
Output schemas completely undocumented. Tools return plain text or JSON without declaring field names, types, or structure. LLMs cannot plan downstream calls or extract data reliably.
Tools 7-9 (walk_directory, list_files, read_file) are inferred from usage in graph.py, not explicitly registered with @mcp.tool() decorator. No visible schema or description.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 34 | 2026-07-28+ | v2 |
Lint Python code for style and correctness issues using ruff. Returns concise findings (file:line + rule + message) such as unused imports/variables, bare excepts, undefined names, and bad practices. Use this to find code-quality and style problems.
Scan Python code for security issues using bandit. Returns concise findings (severity + file:line + issue) for problems such as hardcoded secrets, eval/exec, unsafe subprocess, and weak crypto. Use this to find security risks in the codebase.
Recursively walk a directory and return all files found (inferred from usage in scanner agent)
Write content to a file and auto-generate a PDF report (used by reporter agent to save markdown health report)
No input validation or error recovery guidance. Subprocess errors return raw stderr or timeout messages. LLMs cannot self-correct. Should return 'Path not found. Ensure the directory exists and is readable.' or 'Command timed out after 60s. Try a smaller project or increase timeout.'
Path parameter accepts any string with no validation. Risk of path traversal attacks (e.g., '../../../etc/passwd'). Should validate that path is within allowed project root and reject absolute paths outside scope.