Loading ToolBench
MCP server for Jenkins CI/CD integration, exposing tools to list jobs, trigger builds, retrieve build details, and manage builds.
This server exhibits significant quality gaps across naming, parameter description, schema formalization, and error handling. While tool names are generally verb-based (list_, trigger_, get_, cancel_), many descriptions are vague or generic. Parameter schemas are informally defined as custom ToolProperty classes rather than standard JSON Schema, making them difficult for MCP clients to validate and for LLMs to reason about. Error handling is minimal, most functions return bare error strings without guidance on recovery, retryability, or classification. The server lacks documentation of output schemas, pagination support, and idempotent operation semantics. Of 15 tools, 3 (hello_mcp, get_snippet, save_snippet) are example tools that should be removed. The 12 Jenkins tools show repetitive naming patterns (6 different 'last build' variants) that could be consolidated into a single parameterized tool. Secrets are properly injected via environment variables, which is good, but the server provides no input validation, rate limiting, or audit logging.
Cancel a Jenkins build (stop, kill, or term).
Get console output of a specific Jenkins build job.
Get details of a Jenkins job.
Get details of a specific Jenkins build job.
Get details of the last build for a Jenkins job.
Get details of the last completed build for a Jenkins job.
Get details of the last failed build for a Jenkins job.
Parameter schemas are not defined as JSON Schema but as ad-hoc ToolProperty classes serialized to JSON strings. This violates MCP's strict schema format and makes client-side validation impossible. Clients cannot determine parameter types, required fields, or constraints programmatically.
Six near-identical tools (get_last_build, get_last_completed_build, get_last_failed_build, get_last_stable_build, get_last_success_build, get_last_unsuccessful_build) should be consolidated into a single parameterized tool with a 'status_filter' enum parameter. This reduces cognitive load on LLMs and maintenance burden.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 5 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 32 | - | v1 |
Get details of the last stable build for a Jenkins job.
Get details of the last successful build for a Jenkins job.
Get details of the last unsuccessful build for a Jenkins job.
List Jenkins jobs, optionally filtered by name. This trigger allows you to query a Jenkins server for all jobs, or filter by a substring in the job name.
Trigger a Jenkins build with optional parameters. This trigger allows you to start a Jenkins job, optionally passing build parameters.
No output schema documentation for any tool. LLMs cannot determine what fields to expect in responses, forcing them to guess at structure and limiting their ability to chain tool calls. E.g., does list_jenkins_jobs return job IDs that trigger_jenkins_build accepts?
Error handling is minimal and non-actionable. All error responses are bare exception strings (e.g., 'Error calling Jenkins API: {e}') without guidance on retryability, user-fixable causes, or next steps. No error classification (retryable vs fatal).
Descriptions for generic getter tools are too short and vague. E.g., 'Get details of the last build for a Jenkins job' (56 chars) does not explain what 'details' means (status? duration? logs? parameters?) or when to use it vs get_last_completed_build.
No input validation before API calls. LLMs can pass invalid Jenkins server FQDNs, malformed build names, or out-of-range output limits without clear feedback. Tool should validate and return actionable error messages (e.g., 'Invalid FQDN: must be a valid hostname').
Parameters lack explicit enums and format constraints. E.g., cancel_build's 'action' parameter accepts 'stop, kill, or term' as a description but not as a formal enum constraint. LLMs may guess invalid values like 'terminate' or 'pause'.
No pagination, result limiting, or cursor support. List tools like list_jenkins_jobs do not declare max results or pagination. If a Jenkins instance has thousands of jobs, the response could exceed context limits or timeout.
Example tools (hello_mcp, get_snippet, save_snippet) are included in production tool set. These should be removed or moved to a separate example module.
No idempotency guarantees documented. trigger_jenkins_build is a state-mutating operation, if an LLM retries due to transient failure, will it trigger two builds or deduplicate? The description does not clarify this.
Tool descriptions do not explain prerequisites or dependencies. E.g., trigger_jenkins_build does not mention that the build must exist first (requiring a list_jenkins_jobs call), or that invalid buildParameters will silently fail.