Loading ToolBench
26 tools with complete schemas and descriptions, but quality is uneven. Most tools have adequate naming (verb_noun pattern) and descriptions (100-250 chars), but parameter descriptions are sparse or missing. Many parameters lack type constraints (enums for state/sort fields). Output schemas are not documented. Error handling is present but generic. No tool annotations (readOnlyHint/destructiveHint). Composition is good, tools are single-responsibility and chainable. Security: credentials appear to be injected server-side (no visible API key params), but no audit logging or permission gates visible in code.
Add a comment to an existing issue
Create a new branch in a private GitHub repository [git.dev.hpcnt.com]
Create a new issue in a private GitHub repository [git.dev.hpcnt.com]
Create or update a single file in a private GitHub repository [git.dev.hpcnt.com]
Create a new pull request in a private GitHub repository [git.dev.hpcnt.com]
Create a review on a pull request
No output schemas documented. Tools return data but LLMs cannot plan downstream calls or extract fields reliably. E.g., list_issues returns issues but structure is not declared.
Parameter descriptions missing or minimal. 'page', 'per_page', 'sort', 'direction', 'state' appear across tools with no description text. LLMs cannot infer valid values or constraints.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Agents cannot distinguish safe reads from destructive writes (e.g., merge_pull_request vs get_pull_request). Increases risk of accidental mutations.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Create a new private GitHub repository in your account [git.dev.hpcnt.com]
Fork a private GitHub repository to your account or specified organization [git.dev.hpcnt.com]
Get the contents of a file or directory from a private GitHub repository [git.dev.hpcnt.com]
Get details of a specific issue in a private GitHub repository. [git.dev.hpcnt.com]
Get details of a specific pull request
Get the review comments on a pull request
Get the list of files changed in a pull request
Get the reviews on a pull request
Get the combined status of all status checks for a pull request
Get list of commits of a branch in a private GitHub repository [git.dev.hpcnt.com]
List issues in a private GitHub repository with filtering options [git.dev.hpcnt.com]
List and filter repository pull requests
Merge a pull request
Push multiple files to a private GitHub repository in a single commit [git.dev.hpcnt.com]
Search for code across private GitHub repositories [git.dev.hpcnt.com]
Search for issues and pull requests across private GitHub repositories [git.dev.hpcnt.com]
Search for private GitHub repositories [git.dev.hpcnt.com]
Search for users on GitHub
Update an existing issue in a private GitHub repository [git.dev.hpcnt.com]
Update a pull request branch with the latest changes from the base branch
Enum constraints missing for state/sort/direction fields. 'state' in list_issues accepts 'open|closed|all' but is not declared as enum in schema, LLMs may hallucinate invalid values like 'pending' or 'resolved'.
Error handling is generic. Code shows custom error classes (GitHubError, GitHubRateLimitError) but no recovery guidance in tool descriptions. E.g., 'Rate limit exceeded' does not suggest retry timing or alternative actions.