Loading ToolBench
MCP server for DevBox that provides tools to run commands, check health, read logs, and read contract files with policy-based access control
DevBox MCP has 4 tools with basic schemas and descriptions, but significant gaps prevent production use. Tool names lack action verbs (box-run, box-health, box-read-logs, box-read-contract are noun-heavy). Descriptions exist but are brief (30-80 chars, below the 50-200 char LLM-optimized baseline). Input schemas are present but sparse: box-health has empty input {}, box-run lacks type info for 'args' array elements, box-read-logs has numeric 'tailLines' without min/max bounds. No output schemas documented anywhere. Error handling is minimal, code throws generic errors without recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Parameter descriptions are terse and lack format/constraint details. The server reads from .box/box.yaml and .box/policies.yaml for command allowlisting, which is a security pattern, but this complexity is not surfaced in tool descriptions to guide LLM usage.
Run the health check command defined in .box/box.yaml. Command must be allowlisted in .box/policies.yaml
Read a file from the repository. Path must be within read_paths allowed by .box/policies.yaml
Read log files matching a glob pattern with optional filtering. Respects read_paths policy restrictions
Run a command from .box/box.yaml with optional arguments. Command must be allowlisted in .box/policies.yaml
Tool names lack action verbs. 'box-run', 'box-health', 'box-read-logs', 'box-read-contract' are noun-heavy and do not clearly signal intent. LLMs rely on verb-first naming (run_command, check_health, read_logs, read_file) to infer action before reading descriptions.
No output schemas documented. Tools return text responses but LLMs cannot plan downstream calls or extract structured data without knowing the response format. box-run returns {code, stdout, stderr} but this is not declared in tool metadata.
Parameter constraints missing. 'tailLines' accepts a number but lacks min/max bounds (code enforces max 2000 internally, not exposed). 'glob' pattern has no format guidance. LLMs will pass invalid values without explicit constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 42 | 2026-07-28+ | v2 |
Error handling provides no recovery guidance. Code throws 'Command not allowlisted' and 'Path escapes repo root' but does not suggest next steps (e.g., 'Check .box/policies.yaml for allowed commands'). Errors are not categorized as retryable or user-fixable.
No tool annotations. Tools lack readOnlyHint (box-health, box-read-logs, box-read-contract are read-only) and destructiveHint (box-run can modify state). LLMs cannot distinguish safe from risky operations without explicit hints.