Loading ToolBench
An MCP-integrated GitHub assistant that uses Haystack, Google Gemini, and GitHub MCP tools to analyze repositories, fetch issues, pull requests, commits, and discussions, and suggest automated fixes.
This MCP server has critical definition quality gaps. Only 4 of 8 tools are actually defined in the codebase with visible schemas (get_issues, get_prs, get_commits, get_discussions in github_api_tools.py). The other 4 tools (list_issues, list_pull_requests, list_commits, list_discussions in stream_app.py) are instantiated as MCPTool wrappers pointing to an external GitHub MCP server, their actual schemas and definitions are not visible in this codebase, making them unmeasurable. Tool descriptions are present but generic (10-80 chars), lacking context about WHEN to use each tool vs alternatives. No parameter type information is visible in the JSON schema, only bare strings with minimal descriptions. Error handling is minimal: token validation returns strings like 'Error: ...' but provides no recovery guidance. Tool compositions create confusion: both get_issues and list_issues appear to do the same thing, as do get_prs/list_pull_requests, get_commits/list_commits, get_discussions/list_discussions. This violates the single-responsibility and composition patterns.
Fetch recent commits for a repo 'owner/repo' via GitHub REST API.
Fetch discussions for a repo 'owner/repo' via GitHub REST API (may require repo features and permissions).
Fetch open issues (not PRs) for a repo 'owner/repo' via GitHub REST API.
Fetch open pull requests for a repo 'owner/repo' via GitHub REST API.
List recent commits for a repository (via GitHub MCP server).
List discussions for a repository (via GitHub MCP server).
Four tools (list_issues, list_pull_requests, list_commits, list_discussions) are instantiated as MCPTool wrappers with no visible schema definitions in the codebase. Their actual input/output schemas are delegated to an external GitHub MCP server (https://api.githubcopilot.com/mcp/). Cannot validate or score schema quality.
Duplicate tool definitions: get_issues and list_issues both fetch open issues; get_prs and list_pull_requests both fetch PRs. LLMs will waste reasoning cycles deciding between functionally identical tools. Violates single-responsibility and composition patterns.
Tool descriptions are too generic and lack WHEN/WHY context. 'Fetch open issues' (39 chars) does not explain when an LLM should call this vs list_issues, what the result structure is, or what happens if the repo does not exist. Baselines expect 50-200 chars with explicit context.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 24 | - | v1 |
List open issues for a repository (via GitHub MCP server).
List open pull requests for a repository (via GitHub MCP server).
Parameter 'repo' has only a bare string type and minimal description ('Repository in format owner/repo'). No regex pattern, length constraints, or examples provided. LLMs cannot validate the format without explicit constraints, should use pattern '^[\w-]+/[\w.-]+$' in schema.
No output schema documented for any tool. Callers do not know what fields to expect, breaking downstream tool chaining. E.g., if get_issues returns JSON, does it include 'issue_id', 'id', 'number'? Agents cannot plan follow-up calls without this information.
Error handling is minimal and non-actionable. Tools return strings like 'Error: request failed - <exception>' or 'Error: 401 - <response text>'. No recovery guidance (e.g., 'Token may be expired; check GITHUB_PERSONAL_ACCESS_TOKEN'), no error classification (retryable vs fatal), no suggestions for alternative actions.
GitHub token is accessed from environment variable GITHUB_PERSONAL_ACCESS_TOKEN and passed in HTTP headers. While not exposed as a tool parameter, the token is not properly scoped or rotation-protected. No evidence of scope declaration (e.g., 'read:repo', 'public_repo'). Reduces audit clarity.
No pagination parameters (limit, page, cursor) exposed on get_* tools. Tools return up to 30 issues and 20 commits hardcoded. If a user requests 'all recent PRs', the LLM cannot control result size, risking context window exhaustion.
The apply_code_fix_and_pr() function in stream_app.py creates a PR but is not exposed as a tool. This is a state-modifying operation (writes to repo) that should be wrapped as a tool with a confirmation/dry-run step to prevent accidental destructive actions.