Loading ToolBench
Server has 7 tools with mixed quality. Tools 1-5 are GitLab-specific with good descriptions (100-200 chars, LLM-optimized) and clear parameter schemas. However, tools 6-7 are Jira tools that appear to be duplicates of tools 1-2 with identical names ('list_projects', 'search_issues'), creating severe naming conflicts and confusion. Parameter schemas are present and typed for all tools. Critical issue: tool descriptions for Jira tools claim to be Jira-specific but are registered under the same names as GitLab tools, making them indistinguishable to LLMs. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all being read-only. Error handling guidance is absent, tools do not explain recovery steps on failure.
List merge requests in a GitLab project, filtered by state. Use this when the user asks about open MRs / pull requests, wants to see what's waiting for review, or asks 'what's blocked' on a project. Each MR includes title, state (opened/closed/merged), draft flag, author, branches, and timestamps. IMPORTANT: project_id is a numeric GitLab project ID. You must obtain it by calling list_projects first — never guess it, and never pass a placeholder like 0. If you don't already know the project's ID, call list_projects before this tool. Args: project_id: numeric GitLab project id (get from list_projects). state: one of 'opened' (default), 'closed', 'merged', 'locked', or 'all'. Default 'opened' covers the most common question ('what's pending?').
Get the most recent CI pipelines for a GitLab project. Use this when the user asks 'did the build pass?', 'is CI green?', 'what broke the pipeline?', or wants to see recent pipeline history. Returns each pipeline's status (success/failed/running/pending/canceled), ref (branch), sha, duration, and web url. Pipelines are returned newest-first. The first result is almost always what 'did it pass?' refers to. IMPORTANT: project_id is a numeric GitLab project ID. You must obtain it by calling list_projects first — never guess it, and never pass a placeholder like 0. If you don't already know the project's ID, call list_projects before this tool. Args: project_id: numeric GitLab project id (get from list_projects). limit: how many recent pipelines to return (1-100, default 10).
Summarize a user's recent activity across GitLab. Use this when the user asks what someone has been working on, wants a status update on a colleague, or asks for standup-style summaries. Returns a structured summary: counts of pushes / merge requests opened / issues opened / comments, plus titles of headline activity. This is NOT a list of every event — it's an aggregated summary designed for an agent to reason about. For raw event history, the user should go to the GitLab UI. Args: username: GitLab username (case-sensitive). since: ISO date string (YYYY-MM-DD) for the lower bound. Default is 7 days ago if omitted.
Duplicate tool names: 'list_projects' and 'search_issues' registered twice (GitLab and Jira variants). LLMs cannot distinguish between them; both will be presented as identical tools. This violates the single-responsibility principle and creates ambiguity.
No tool annotations present. All 7 tools are read-only (risk=READ_ONLY declared), but readOnlyHint is not set in tool definitions. This prevents LLMs from understanding which tools are safe to call without confirmation.
No error recovery guidance. Tool descriptions do not explain what to do if a call fails (e.g., 'If project_id is invalid, call list_projects first'). Error responses are not documented.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | <=2025-11-25 | v2 |
List all GitLab projects the configured user belongs to. Use this when the user asks what projects/repos/codebases exist, wants an overview of what's available, or before drilling into any specific project. Returns each project's id (needed for other tools), name, namespace path, default branch, and last-activity timestamp. Most other tools require a project_id, so this is typically the first tool to call when the user names a project ambiguously (e.g. 'the order service' rather than giving an id).
List all Jira projects the configured account can access. Returns each project's key (e.g. 'SCRUM', 'AM'), numeric id, and name. Call this FIRST whenever you need a project key for search_issues — never guess a project key. Most questions about a specific project start here to resolve its key. Examples: 'what projects are there?', 'list my Jira projects'.
Search issues in a GitLab project by keyword. Use this when the user asks about bugs, feature requests, or any work items in a project — and wants to filter by topic. Examples: 'are there any open performance issues?', 'find bugs related to payments'. For 'show me ALL issues' (no keyword), pass an empty query string. IMPORTANT: project_id is a numeric GitLab project ID. You must obtain it by calling list_projects first — never guess it, and never pass a placeholder like 0. If you don't already know the project's ID, call list_projects before this tool. Args: project_id: numeric GitLab project id — obtain via list_projects. query: free-text search; matches title and description. state: 'opened', 'closed', or 'all' (default 'all').
Search issues within a Jira project by free-text keyword. Use this for questions about bugs, tasks, stories, or any work items in a project. Returns each matching issue's key (e.g. 'SCRUM-12'), summary, status, and type. IMPORTANT: project_key is a short alphanumeric Jira project key like 'SCRUM' or 'AM' — NOT a numeric id, and NOT a guess. Obtain it by calling list_projects first if you don't already know it. For all issues in a project, pass an empty string as the query. Examples: 'find login bugs in the mobile project', 'what crash issues are there?', 'show all issues in SCRUM'. Args: project_key: short Jira project key (obtain via list_projects). query: free-text keyword; matches summary, description, comments. Pass '' for all issues.
Parameter 'state' in get_merge_requests and search_issues lacks enum constraint in schema. Description lists valid values ('opened', 'closed', 'merged', etc.) but schema does not enforce them, allowing LLMs to pass invalid values.
Output schemas not documented. Tool descriptions do not specify what fields are returned (e.g., get_merge_requests returns 'title, state, draft, author, branches, timestamps' but no formal schema). LLMs cannot plan downstream tool calls without knowing response structure.