Loading ToolBench
Model Context Protocol server for Azure DevOps — boards, repos, pipelines, and projects with governance controls (access modes, project scoping, protected projects, delete gating, confirmation, dry-run, audit).
Strong foundation with 21 well-named tools, comprehensive descriptions (avg 150+ chars), and explicit input schemas with types. Tool names follow verb_noun pattern (list_*, get_*, create_*, update_*, delete_*, run_*). All parameters have type definitions and descriptions. Output schemas are documented in descriptions. Security controls (allowlist, protected projects, confirmation, dry-run, audit) are well-integrated. Main gaps: output field documentation is implicit rather than explicit (no formal response schema definitions in code), some parameter descriptions lack format/constraint details (e.g., WIQL syntax), and error recovery guidance is minimal. Composition is clean, each tool has one responsibility, and tool chains are well-designed (e.g., list_projects → get_project → list_teams → list_team_members).
Create a new project. Requires admin mode. Provide a process template id (see list_processes).
Open a pull request from a source branch into a target branch.
Create a work item (e.g. Bug, Task, User Story) with a title and optional description.
Permanently delete a project (all repos, work items, pipelines). Requires admin mode, AZDO_ALLOW_DELETE=true, and a typed confirmation equal to the project name. Irreversible.
Fetch a single build / pipeline run by id, with its status, result, and timing.
Fetch the full details (id, description, visibility, state, capabilities) of a single project.
Output schemas are documented only in tool descriptions, not as formal JSON Schema definitions. LLMs cannot parse implicit documentation to understand response structure for chaining.
query_work_items parameter 'wiql' lacks format/syntax guidance. Description mentions WIQL but does not explain valid syntax, operators, or common mistakes. LLMs may generate invalid queries.
Error handling in tool descriptions is minimal. No guidance on retryability, user-fixable errors, or recovery steps. toErrorResult() in server.ts truncates API errors to 800 chars, losing context.
create_work_item and update_work_item accept 'fields' as an untyped object. LLMs cannot infer valid field names or types without documentation of the field map structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
Fetch a single pull request by id, including its source/target branches, status, and reviewers.
Fetch a single work item by id, including all of its fields (title, state, assignee, tags, …).
List the branches (refs/heads) of a Git repository, with the commit each one points at.
List recent builds / pipeline runs in a project, most recent first, with their status and result.
List a team's iterations (sprints), each with its path and start/finish dates.
List the pipelines defined in a project, with their ids and names. Use a pipeline id with run_pipeline.
List the organization's process templates (Agile, Scrum, CMMI, or inherited) with their ids. Pass one of these ids as the process when creating a project.
List the projects in the Azure DevOps organization that pass the allowlist. Returns each project's id, name, state, and whether it is marked protected (readable but never mutated). Start here to discover valid project names.
List pull requests in a repository, optionally filtered by status (active by default).
List the Git repositories in a project, with their ids, names, and default branches.
List the members (identities: display name and unique name) of a specific team within a project.
List the teams defined in a project, with their ids and names. Use a team id/name with the team-scoped tools.
Run a Work Item Query Language (WIQL) query against a project and return the matching work-item ids (fetch fields with get_work_item). Example: SELECT [System.Id] FROM WorkItems WHERE [System.State] = 'Active'.
Queue a run of a pipeline, optionally on a specific branch.
Update fields of a work item (title, state, assignee, or arbitrary field paths).
Pagination is not implemented for list_* tools. list_projects, list_teams, list_repositories, etc. may return unbounded results, risking context window exhaustion.