Loading ToolBench
MCP server for code review automation with GitHub PR integration, semantic search of past reviews, and review storage with embeddings
Four tools with basic verb-noun naming (get_, comment_, search_, store_) but significant quality gaps. All tools have descriptions (10-60 chars, below the 194-char baseline), but lack depth on WHEN to use them and prerequisites. Input schemas present but minimal: only 1-2 params per tool with type strings and short descriptions. No output schemas documented. No error handling guidance, no recovery paths, no parameter constraints (enums, ranges). Tools lack idempotency hints and permission declarations. Composition is reasonable (each tool has one job), but chaining is weak, store_review requires diff_text as input even though get_pr_diff already fetches it, forcing the agent to manage intermediate state.
Post a review comment on the pull request.
Fetch the unified diff of a GitHub pull request.
Search for top 3 similar past reviews using pgvector.
Store the diff, embedding, and review in the database. Returns the review ID.
No output schemas documented. LLMs cannot plan downstream calls or extract structured data. get_pr_diff returns 'str' (raw diff text), comment_on_pr returns 'str' (URL), search_similar_reviews returns 'str' (formatted text), store_review returns 'str' (review ID). All should declare field names, types, and structure.
Descriptions are too brief (30-60 chars vs 194-char baseline). Missing WHEN to use each tool and prerequisites. 'Fetch the unified diff' does not explain when to call this vs search_similar_reviews. 'Post a review comment' does not clarify if this is a PR review or issue comment.
No error handling guidance. Tools call external APIs (GitHub, Supabase) and will fail (auth, network, rate limits, invalid PR URL). No recovery hints. E.g., parse_pr_url raises ValueError but LLM sees no guidance on how to fix malformed URLs.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | 2026-07-28+ | v2 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). comment_on_pr and store_review are destructive (write state) but LLM has no signal. get_pr_diff and search_similar_reviews are read-only but unmarked. Agents cannot reason about safety or retry logic.
Weak tool chaining. store_review requires diff_text as a parameter, but get_pr_diff already fetches it. Agent must manually pass the diff between calls, increasing context and error risk. Consider making store_review accept only pr_url and fetch diff internally, or return diff from get_pr_diff in structured form.