Loading ToolBench
MCP server for interacting with Github
The server has 21 well-named tools with complete input schemas and descriptions for each parameter. Tool naming follows verb_noun conventions consistently (get_issue, create_issue, search_issues, list_pull_requests, etc.). Most descriptions are present and adequate (50-150 chars typical), though some are minimal. All parameters are typed with zod schemas and include descriptions. A significant weakness: tool descriptions lack action-oriented context (WHEN to use, prerequisites, what it returns) and error handling guidance is minimal. Output schemas are not formally documented, responses are serialized as JSON but the structure is not declared in parameter schemas. Pagination is present in listing tools but not consistently documented. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite the server having clear risk classifications (WRITE, IRREVERSIBLE, READ_ONLY). Per-tool analysis follows.
Add a comment to a specific issue in a GitHub repository.
Create a new issue in a GitHub repository.
Create or update a single file in a GitHub repository. If updating an existing file, you must provide the current SHA of the file (the full 40-character SHA, not a shortened version).
Get details for a commit from a GitHub repository
Get details of a specific issue in a GitHub repository.
Get comments for a specific issue in a GitHub repository.
Tool descriptions are minimal and lack action-oriented context. Most descriptions state WHAT but not WHEN to use or WHERE it fits in workflows. Example: 'Update an existing issue in a GitHub repository' does not explain whether to use update_issue or add_issue_comment for status changes.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk classifications present in the data. merge_pull_request is marked IRREVERSIBLE but has no destructiveHint; all READ_ONLY tools lack readOnlyHint. These annotations enable LLM planning.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 57 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 55 | - | v1 |
Get details of a specific pull request in a GitHub repository.
Get the files changed in a specific pull request.
Get the status of a specific pull request.
Get detailed information about a GitHub repository including README and file structure
List branches in a GitHub repository
Get list of commits of a branch in a GitHub repository
List issues in a GitHub repository.
List pull requests in a GitHub repository.
Merge a pull request in a GitHub repository.
Search for code across GitHub repositories. Returns a concise list with file paths and repositories. Use 'get_file_contents' for full file content.
Search for issues in GitHub repositories.
Search for GitHub repositories. Returns a concise list with essential information. Use 'get_repository' for detailed information about a specific repository.
Search for GitHub users.
Update an existing issue in a GitHub repository.
Update an existing pull request in a GitHub repository.
Output schemas are not formally documented. Tools return JSON via { type: 'text', text: JSON.stringify(response.data) } but the structure is not declared. LLMs cannot verify expected fields exist. Example: search_issues returns 'items' and 'total_count' but this is not in a formal schema.
Error handling returns raw error messages ('Error: {e.message}') with no guidance for recovery. Examples: 'Error: Not Found' does not tell the LLM to retry with a different query or call search_* first. Missing error categorization (retryable vs user-fixable vs fatal) and actionable suggestions.
create_or_update_file description says 'If updating an existing file, you must provide the current SHA' but the input schema does NOT include a 'sha' parameter. The tool cannot accept the SHA, making the instruction impossible to follow.
Several tools with enum parameters use inconsistent naming. search_issues accepts 'reactions-+1' (with special chars) while others use snake_case. The '+1' enum value is error-prone for LLMs to generate correctly.
Pagination parameters (page, per_page) are present in listing tools but not consistently documented with limits. per_page max=100 is mentioned in some tools but not all (e.g., list_branches does not document the max).
Response filtering and stripping is inconsistent. search_issues returns formatted markdown with a count ('Found X total results') while get_issue returns raw JSON. This requires the LLM to handle heterogeneous response formats, increasing errors.