Loading ToolBench
MCP server for GitHub API operations including branch management and commit listing
Server has 5 tools with basic schemas and descriptions, but significant gaps in parameter documentation, output schema clarity, and error handling. Tool names follow verb_noun convention (list_commits, create_branch, get_branch_sha, create_branch_from_ref, update_branch), which is good. However, descriptions are minimal (10-50 chars), parameter descriptions are sparse or missing, and output schemas are not documented. No error handling guidance, no recovery paths, and no indication of which operations are destructive. The codebase shows Pydantic models for validation but these are not exposed in tool schemas. Missing idempotency hints and confirmation patterns for write operations.
Create a new branch in a repository
Create a new branch from an existing branch or default branch
Get the SHA of a specific branch
List commits for a repository
Update a branch to point to a new SHA
Descriptions are too short (10-50 chars) and lack context. 'List commits for a repository' does not explain when to use this vs other tools, what fields are returned, or pagination behavior. LLMs cannot reliably select this tool without richer guidance.
Output schemas are not documented. Callers cannot know what fields to expect from list_commits (does it return commit objects with sha, message, author?), create_branch (does it return the new ref object?), or get_branch_sha (does it return a string or an object?). This forces LLMs to guess and wastes tokens on trial-and-error.
Write operations (create_branch, create_branch_from_ref, update_branch) lack destructive hints and confirmation patterns. No indication that these are irreversible or that agents should confirm before executing. Agents may accidentally create/update branches without user approval.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Parameter descriptions are missing or minimal. 'page' and 'per_page' in list_commits lack range constraints (e.g., 1-100). 'sha' in create_branch is described as 'SHA of the commit' but does not clarify format or validation. 'from_branch' in create_branch_from_ref says 'optional, uses default if not provided' but does not explain which default or how to discover it.
No error handling guidance. Code defines GitHubError, GitHubAuthenticationError, GitHubRateLimitError, etc., but tools do not document what errors they can raise or how to recover. LLMs receive raw exceptions with no actionable next steps.