Loading ToolBench
11 tools with complete schemas and descriptions, but significant quality gaps. All tools have descriptions (10-200 chars, mostly adequate) and input schemas with types. However, many descriptions lack actionable context for LLM selection. Parameter descriptions are present but often generic. No output schemas documented. Error handling guidance absent. No tool annotations (readOnlyHint/destructiveHint). register_app and finish_bitrise_app descriptions are verbose but lack clarity on prerequisites and failure modes. update_app requires all 5 params but should have defaults for optional fields. No pagination guidance for list_apps despite returning potentially large result sets.
Delete an app from Bitrise. When deleting apps belonging to multiple workspaces always confirm that which workspaces' apps the user wants to delete.
Finish the setup of a Bitrise app. If this is successful, a build can be triggered via trigger_bitrise_build. If you have access to the repository, decide the project type, the stack ID, and the config to use, based on https://stacks.bitrise.io/, and the config should be also based on the projec type.
Get the details of a specific app.
Get the current Bitrise YML config file of a specified Bitrise app.
List all the apps available for the authenticated account.
List the branches with existing builds of an app's repository.
No output schemas documented. Tools return raw API responses as strings without structure hints. LLMs cannot plan downstream calls or extract required fields (e.g., app_slug from list_apps for use in get_app).
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Destructive tools (delete_app) and write tools lack explicit risk markers. LLMs cannot distinguish safe reads from irreversible operations without reading descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
Add a new app to Bitrise. After this app should be finished on order to be registered completely on Bitrise (via the finish_bitrise_app tool). Before doing this step, try understanding the repository details from the repository URL. This is a two-step process. First, you register the app with the Bitrise API, and then you finish the setup. The first step creates a new app in Bitrise, and the second step configures it with the necessary settings. If the user has multiple workspaces, always prompt the user to choose which one you should use. Don't prompt the user for finishing the app, just do it automatically.
Add an SSH-key to a specific app.
Register an incoming webhook for a specific application.
Update an app.
Update the Bitrise YML config file of a specified Bitrise app.
update_app requires all 5 parameters (is_public, project_type, provider, repo_url) with no defaults. Partial updates should allow selective field modification. Current design forces LLM to provide all fields even when only one changes.
list_apps and list_branches lack pagination result limits and total count guidance. No mention of max results returned or when to use next/limit parameters. Large result sets could exhaust context window.
Error handling absent. No guidance on retryable vs fatal errors, no recovery suggestions. call_api() returns raw response.text without error classification or actionable messages for LLM.