Loading ToolBench
Model Context Protocol server for devpipe - helps AI assistants interact with devpipe to manage development workflows
Server has 25 tools with consistent naming (verb_noun pattern) and descriptions present for all. However, output schemas are entirely undocumented, no tool declares what fields it returns, forcing LLMs to guess. Parameter descriptions are present but often generic (e.g., 'Path to config.toml file' repeated 15+ times). Error handling is absent: no recovery guidance, no categorization of retryable vs fatal errors. Tool composition is reasonable (single responsibility mostly observed), but many tools lack actionable parameter constraints (enums, ranges, formats). No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels. Baseline: 549 production tools average 194-char descriptions; this server averages ~100 chars, often too terse.
Analyze correlation between file changes and task failures.
Analyze project directory to detect technologies and suggest missing tasks.
Check if devpipe is installed and get version information.
Compare results between two devpipe runs.
Create a new devpipe config.toml file with initial tasks based on detected technologies.
Detect tasks that have inconsistent pass/fail results across runs.
Detect tasks with significant performance degradation across runs.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. All 25 tools lack return type declarations.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). run_pipeline and create_config are WRITE operations but unmarked; agents cannot distinguish safe from risky calls.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Generate CI/CD configuration files (GitHub Actions, GitLab CI, etc.) based on devpipe config.
Generate task configuration from template for a specific technology and task type. Use technology="phase" to create phase headers.
Get list of changed files since a git reference.
Extract aggregated data from summary.json or the HTML dashboard, including overall run statistics and metrics.
Get current git status including staged and unstaged changes.
Get results and metadata from the most recent devpipe run, including task results, duration, and success status.
Get aggregated metrics summary from all runs.
Get overall pipeline health metrics and status.
Get list of recently failed tasks across runs.
Get execution history for a specific task across multiple runs.
Analyze which tasks would be affected by changes to specific files.
Parse and list all tasks from a devpipe config.toml file. Shows task IDs, names, types, commands, and enabled status.
List tasks using devpipe list --verbose command. Shows task execution statistics and averages.
Parse JUnit or SARIF metrics from a devpipe run to analyze test results or security findings.
Predict which tasks will be affected by proposed changes.
Execute devpipe with specified configuration and flags. Runs the development pipeline and returns results.
Validate one or more devpipe config.toml files for syntax and structure errors.
Read logs from a specific task or the entire pipeline from the most recent run.
Error handling absent. No recovery guidance, no error categorization (retryable vs fatal), no actionable error messages. Agents cannot self-correct on failures.
Parameter descriptions are generic and repetitive. 'Path to config.toml file' appears 15+ times; no guidance on format, constraints, or when to omit. Descriptions average ~100 chars vs production baseline of 194 chars.
No pagination or result limits documented. Tools like list_tasks, get_task_history, get_recent_failures may return unbounded lists, risking context window exhaustion.