Loading ToolBench
Entity-level code review server for analyzing code changes at the entity level (functions, classes, etc.) using semantic analysis and risk scoring. Provides tools for triaging changes, inspecting entities, predicting at-risk code, and posting reviews to GitHub.
The server has 10 well-structured tools with consistent naming (all prefixed with 'inspect_') and complete input schemas using JSON Schema via schemars. Descriptions are present and reasonably detailed (averaging ~150 chars), explaining the primary use case and when to call each tool. However, critical gaps reduce the score: (1) output schemas are completely undocumented, no descriptions of what each tool returns, (2) several parameters lack constraints (e.g., min_risk accepts a free-form string 'low'|'medium'|'high'|'critical' but is not declared as an enum), (3) error handling guidance is absent, tools provide no hints for recovery or retry strategies, and (4) the comments array in inspect_post_review has nested fields (path, line, body, start_line) but the ReviewComment struct lacks full descriptions for nested properties. The tools are READ_ONLY by design except inspect_post_review (WRITE), which is appropriate, but no tool descriptions explicitly state immutability or side effects. Parameter descriptions are concise but could better explain constraints (e.g., repo_path should clarify 'absolute path' and GitHub context for remote tools).
Drill into a single entity to see full details including before/after content, dependents, and dependencies. Use after inspect_triage to understand a specific high-risk entity.
Scope triage to a specific file. Returns only changed entities in that file, useful for narrowing focus on file-level changes.
Get all entities in a logical change group. Groups are formed by dependency edges between changed entities. Use after inspect_triage to understand related changes.
Post a structured review with an overall summary and per-file comments to a GitHub PR. Comments can be multi-line. Respects GitHub's diff constraints (only commentable lines).
Predict which unchanged entities are at risk of breaking due to the changes. Returns entities sorted by risk of being affected, with the changed entities they depend on.
Analyze a GitHub PR by number without cloning. Fetches the PR metadata and file diffs directly from GitHub. Returns the same triage output as inspect_triage.
Output schemas are completely undocumented. No tool description explains what fields are returned, what data structures are nested, or what IDs/references are included for chaining. This forces LLMs to guess at response structure and breaks tool composition patterns.
min_risk parameter accepts a free-form string without enum constraint. Tools should declare 'low|medium|high|critical' as an enum in the JSON Schema, not rely on description alone. LLMs will hallucinate invalid values like 'urgent' or 'critical-high'.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 65 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 36 | - | v1 |
Get a risk distribution visualization: files with entity counts by risk level. Useful for understanding which files have the most problematic changes.
Search for a text pattern in the PR files and optionally across the entire codebase via GitHub Code Search. Useful for finding usages of changed entities.
Get summary statistics for the change: counts by risk level, change types, and timing information.
Run entity-level code review triage. Returns a compact summary of all changed entities sorted by risk score, with classification, blast radius, and logical grouping. This is the primary entry point for understanding what changed and where to focus review effort.
Error handling is absent. No tool description explains what errors can occur, whether they are retryable, or what the LLM should do next. For example, if inspect_triage fails on an invalid repo_path, the tool provides no recovery hint (e.g., 'Check that repo_path is absolute and the repository is initialized').
Parameter constraints are implicit, not declared. repo_path and file_path accept arbitrary strings without min/max length, path validation, or regex patterns. LLMs may pass '../../../etc/passwd' or other malformed paths without validation.
ReviewComment nested struct in inspect_post_review lacks full field descriptions. The 'comments' parameter description says 'Each has: path, line, body, start_line' in prose, but path, line, and start_line should have individual enum/type/constraint annotations in the schema.
Tool descriptions do not explicitly state whether the tool modifies state. inspect_post_review clearly performs a write (WRITE risk rating), but tool descriptions should state 'This tool posts a review to GitHub (side effect: irreversible)' so LLMs know it requires confirmation or dry-run support.
No tool composition hints. Tool descriptions should explain which tools are entry points (inspect_triage for discovery) and which follow up (inspect_entity to drill down). Descriptions like 'Use after inspect_triage' are hints but lack systematic dependency documentation.