Loading ToolBench
A Jenkins plugin that implements the Model Context Protocol (MCP) server, enabling AI/LLM clients to interact with Jenkins through standardized tools for job management, build execution, logging, and CI/CD operations.
This Jenkins MCP server exhibits significant quality gaps that prevent production recommendation. Of 22 tools, most lack detailed descriptions and have minimal parameter documentation. Tool names follow verb-noun conventions (e.g., getBuildLog, triggerBuild), which is positive. However, descriptions average ~50 chars, well below the 194-char baseline for A+ tools. Critically, many parameters lack descriptions entirely (e.g., whoAmI has zero parameters documented; sayHello's 'name' parameter has minimal guidance). The plugin-style definition (Java test files) suggests tools are inferred rather than explicitly registered with full schemas, applying the hard cap of 50 for inferred definitions. Only 4 tools (sayHello, testInt, testWithError, getBuildLog) show explicit input schemas; the remainder appear to be test fixtures. Error handling is underdeveloped, no actionable recovery guidance in descriptions. No evidence of pagination support for list operations (getJobs has no limit/offset params). Destructive tools (rebuildBuild, replayBuild, triggerBuild, updateBuild) lack confirmation patterns or dry-run support, violating critical safety patterns.
Find jobs that use a specific SCM repository URL
Get details of a specific build
Get the changesets (commits) for a specific build
Get the full build log for a specific build
Get the SCM details for a specific build
Get flaky test failures for a build
Get details of a specific job
Tool definitions inferred from test files rather than explicitly registered with production metadata. Per hard scoring rules, caps per-tool scores at 50 when registration is not visible in actual tool registration code.
Descriptions severely under-specified. Most tool descriptions (13/22) are under 50 characters, far below the 194-char baseline for production tools. Examples: whoAmI='Get information about the current user' (42 chars), getStatus='Get the overall status of Jenkins' (35 chars). LLMs cannot distinguish when to select these tools vs similar alternatives.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 65 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
Get the Source Control Management (SCM) configuration for a job
Get a list of all jobs in Jenkins
Get information about a queued build item
Get replay scripts for a build
Get the overall status of Jenkins
Get test results for a build
Rebuild a specific build with the same parameters
Replay a build with custom Groovy script modifications
Say hello to someone
Search for log lines matching a pattern
Test tool returning an integer
Test tool that throws an error
Trigger a build for a specific job with optional parameters
Update build properties like description and result
Get information about the current user
Parameter descriptions almost entirely absent. whoAmI has empty parameters object but no descriptions. getJobs has zero parameters but no indication of filtering, pagination, or sorting options. sayHello's 'name' parameter has minimal guidance.
No pagination support on list operations. getJobs returns 'a list of all jobs' with no limit, offset, or page_size parameters. Per pattern:paginated-result, large result sets blow context windows. No evidence of cap enforcement or next_cursor support.
Destructive operations lack confirmation or dry-run patterns. rebuildBuild, replayBuild, triggerBuild, updateBuild are marked DESTRUCTIVE but have no pre-execution confirmation, dry-run mode, or user approval flow. Per pattern:confirmation-request, agents make mistakes, these tools risk unintended CI/CD impacts without safety gates.
No actionable error guidance. No evidence in descriptions of what to do when a tool fails. Per pattern:recovery-guide, errors should advise: 'Job not found, try searching with a partial name' or 'Build is still queued, wait and retry'. Current descriptions do not address failure modes.
Output schemas not documented. Tool descriptions do not specify what fields are returned. For example, getBuild description says 'Get details of a specific build' but does not enumerate: build_id, status, duration, logs_url, artifacts, etc. LLMs cannot plan downstream tool calls without knowing what to extract.
Generic tool name 'testWithError' does not follow verb_noun convention. Name is vague about what it tests or why to call it. Suggests test fixture rather than production tool, should be removed or clarified (e.g., 'trigger_error_scenario', 'validate_error_handling').
Test tools in production namespace. sayHello, testInt, testWithError appear to be test fixtures, not real Jenkins operations. These should be removed or isolated to a dev/test namespace. Pollutes tool catalog and confuses agent tool selection.
No input validation constraints documented. Parameters like jobName, buildNumber accept freeform strings/integers with no validation hints. Per pattern:constrained-input, descriptions should specify: 'buildNumber: integer >= 1', 'jobName: 1-255 chars, no special chars'. Allows agents to pass invalid values requiring retry.