Loading ToolBench
Dev server supervisor for AI coders. Manages dev server processes, monitors health, detects reloads, and auto-recovers from crashes.
DevPilot MCP server demonstrates solid tool design with clear, action-oriented naming (devpilot_status, devpilot_run, devpilot_changed) and comprehensive descriptions (avg 180 chars). All 8 tools have documented input schemas with type definitions and parameter descriptions. However, output schemas are not explicitly documented in the source code, only inferred from docstrings. Error handling guidance is minimal; tools lack recovery hints and actionable error messages. Tool composition is strong (each tool has one responsibility), but some parameters could benefit from enum constraints (e.g., service_type accepts 'backend'|'frontend' but lacks formal enum declaration). Security considerations are present (no secrets in params) but not explicitly documented.
Attach to an already-running dev server for health monitoring. Discovers the process by port but does NOT own or restart it. Use when the server was started externally.
Report a file change and check if the dev server reloaded successfully. Call this AFTER editing a file. Returns which service was affected, reload status (reloaded/reload_failed/timeout), and health. This is the primary feedback loop for AI coders.
Stop all managed services (alias for stop --all).
Auto-detect project structure and generate .devpilot.yaml config. Scans for pyproject.toml, requirements.txt, and package.json to detect frameworks (FastAPI, Flask, Django, Vite, Next.js, CRA) and generates appropriate service configuration.
Start and manage a dev server process. Spawns the process, captures stdout, detects reload patterns, monitors health, and auto-recovers from crashes. Supports FastAPI, Flask, Django, Vite, Next.js, and CRA with auto-detected defaults.
Check health status of dev server services. Returns live health check results including status, response time, and configuration for all registered services or a specific one.
Output schemas not explicitly documented. Tool descriptions mention return types (e.g., 'Returns live health check results') but JSON Schema output definitions are absent from source code. LLMs cannot reliably parse responses or plan downstream tool chains without documented output structure.
Enum constraints not formally declared. Parameters like service_type ('backend'|'frontend') and reload_patterns are described as accepting specific values but lack JSON Schema enum declarations. LLMs may hallucinate invalid values.
Error handling lacks recovery guidance. Tool descriptions do not explain what to do if a service fails to reload, a port is already in use, or a process crashes. Errors should guide the LLM toward corrective actions (e.g., 'Port 8000 already in use. Try devpilot_stop() first or specify a different port.').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
Gracefully stop managed dev server services. Only stops services that devpilot started. Never kills processes it didn't start.
Start all services defined in .devpilot.yaml. Reads the project config and starts each service under supervision with auto-recovery.
Tool names 'devpilot_up' and 'devpilot_down' are vague. 'up' and 'down' do not clearly convey 'start all services' and 'stop all services'. Better names: 'devpilot_start_all' and 'devpilot_stop_all' would be self-documenting.
Mutually exclusive parameters not documented. devpilot_stop accepts both 'name' and 'stop_all' but does not explicitly state they are mutually exclusive. LLMs may pass both, causing ambiguous behavior.