Loading ToolBench
16 tools with complete schemas and descriptions, but quality is uneven. Most tools have adequate naming (verb_noun pattern) and parameter descriptions, but descriptions are often generic (50-100 chars, below the 194-char baseline). Parameter schemas are well-structured with types and enums where appropriate. Critical gaps: no output schemas documented, no error handling guidance, no idempotency markers on write operations, and descriptions lack WHEN/WHY context that LLMs need for tool selection. Composition is reasonable, tools are single-purpose, but some parameter relationships undocumented (e.g., alertType vs states vs severities in advsec_get_alerts).
Get detailed information about a specific Advanced Security alert.
Retrieve Advanced Security alerts for a repository.
Retrieves a list of builds for a given project.
Get the changes associated with a specific build.
Retrieves a list of revisions for a specific build definition.
Retrieves a list of build definitions for a given project.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields. E.g., advsec_get_alerts returns alert objects but structure is undocumented.
Descriptions are generic and lack WHEN/WHY context. E.g., 'Retrieves a list of builds' (55 chars) vs. 'Retrieves a list of builds for a given project. Use this to find builds by definition, status, or time range before querying logs or changes.' (194 chars baseline). LLMs cannot distinguish when to call build_get_builds vs build_get_definitions.
Write operations (build_run_build, build_update_build_stage) lack error handling guidance and idempotency markers. No indication of whether retries are safe or if they risk duplicate side effects (e.g., queuing the same build twice).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Retrieves the logs for a specific build.
Get a specific build log by log ID.
Get the status of a specific build.
Queue a new build for a given build definition.
Update the stage of a specific build.
Retrieve Azure DevOps identity IDs for a provided search filter.
Retrieve a list of teams for the specified Azure DevOps project.
Retrieve a list of projects in your Azure DevOps organization.
Retrieves list of release definitions for a given project.
Retrieves a list of releases for a given project.
Parameter relationships undocumented. E.g., advsec_get_alerts accepts alertType, states, severities, ruleId, ruleName, toolName, and ref, but no guidance on which combinations are valid or mutually exclusive. LLMs may pass conflicting filters.
build_update_build_stage has vague schema. 'stageUpdateType' parameter lacks enum or description of valid values. LLMs cannot determine what to pass.