Loading ToolBench
A read-only Model Context Protocol server for GitHub that allows Claude to query repositories, commits, branches, pull requests, issues, and file contents via OAuth-authenticated access to the GitHub REST API.
Seven read-only tools with consistent verb-noun naming (list_*, get_*, search_*). All tools have descriptions (avg 150 chars, within baseline 34-392). Input schemas present with types and descriptions. However, output schemas are not documented in the source, LLMs cannot predict return structure. Parameter descriptions are good but lack explicit format/constraint details (e.g., 'repo' format is documented as 'owner/name' but not enforced via regex or enum). Error handling is minimal, no recovery guidance or actionable error messages visible. Security is strong (read-only, no secrets in params), but composition could be tighter (e.g., search_code lacks pagination despite potentially large result sets).
Read a file or list a directory's contents at a given path and ref (e.g. README.md, a roadmap doc, or a directory listing). Paths are repo-relative: absolute paths (leading '/') and traversal segments ('.'/'..') are rejected. Use an empty path to list the repo root.
List branches on a repo. Unmerged branches are a reasonable proxy for in-flight work when PR data isn't otherwise available.
List recent commits on a repo/branch, most recent first. Use this to check whether work described in a task or PR is already merged before creating duplicate work.
List issues on a repo (open by default). Pull requests are excluded from the results.
List pull requests on a repo (open by default).
Output schemas not documented. LLMs cannot predict return structure (fields, types, pagination format). Baseline requires 100% of A+ tools to have documented return types.
search_code lacks pagination (limit/offset/cursor). GitHub code search can return hundreds of results; unbounded responses blow context windows. Baseline pattern requires paginated results.
No error recovery guidance. Error responses do not tell LLMs what to do next (e.g., 'Repo not found. Call list_repos() to see available repos.'). Baseline requires actionable error messages.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
List the repositories this connector's GitHub token can read (most recently pushed first). Call this first if unsure what's available.
Search code within a single repo (GitHub code search syntax). The search is already scoped to the given repo, so queries may not contain their own 'repo:', 'org:', or 'user:' qualifiers -- those are rejected.
Parameter constraints not enforced. 'repo' format documented as 'owner/name' but no regex pattern or validation. 'query' in search_code rejects 'repo:' qualifiers but no validation error message shown. Baseline requires explicit format constraints.
list_* tools lack explicit result limits in descriptions. Baseline requires stating the limit (e.g., 'Returns up to 50 items') to prevent context window exhaustion.