Loading ToolBench
MCP server that exposes guarded Node-RED tools to CLI agents (Claude Code, Codex, Gemini CLI) for agentic control of Node-RED flows without API keys
FlowPilot MCP server has 30 tools with significant definition gaps. Most tools lack complete input schemas visible in source (only parameter names/descriptions provided, no explicit JSON Schema type definitions). Descriptions are present but often generic (10-50 chars), below the 50-200 char baseline for LLM-optimized tools. No output schemas documented. Error handling is minimal, tools return success/failure without recovery guidance. Tool naming is reasonable (verb_noun pattern mostly followed), but composition issues exist: multiple tools handle similar concerns (e.g., nodered_deploy_patch, nodered_rollback, nodered_restart all modify state without clear distinctions). Security-sensitive tools (package_install_node, local_run_allowed_command) lack permission gates or audit trail declarations. Per-tool analysis shows 15 tools with descriptions under 50 chars, 8 tools with no visible schema enforcement.
Generate constrained Function node JS.
Static checks: syntax, loops, forbidden APIs, secret leak.
Persist a proposed Node-RED patch draft. Include only changed/new nodes plus any new tab/config nodes; existing tab ids in z are allowed.
Create a draft Node-RED manual-control flow using symbolic command inject buttons.
Diff a draft vs current live flows.
Inject test messages, compare expected outputs.
Run 5 validation passes against a draft.
Descriptions too short (10-50 chars) for LLM selection. Baseline is 50-200 chars. Tools like nodered_get_flow_state ('Runtime state (started/stopped).'), nodered_restart ('Restart Node-RED runtime.'), and local_read_allowed_file ('Read a whitelisted local file.') lack context on WHEN to use them vs similar tools.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields. E.g., nodered_deploy_patch returns success/failure but does not document what fields the response contains or what IDs are available for follow-up calls.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
Read a whitelisted local file.
Run a whitelisted shell command.
Write a whitelisted local file.
Post-deploy health probe.
Capture current flows as a restorable snapshot.
Deploy a validated draft by id through snapshot, merge, health-check and rollback safeguards. Stops affected nodes.
Runtime diagnostics report.
Read one flow tab by id.
Runtime state (started/stopped).
Read active flows. Default view=summary returns compact tabs/nodes for agents; use view=full only when exact deploy JSON is required.
Editor/runtime settings (safe subset).
Installed node modules + node types.
Recent runtime logs buffer.
Restart Node-RED runtime.
Restore a snapshot.
Check admin auth, https, exposed editor, dangerous nodes (Threat 3).
Inspect npm node package reputation/capabilities.
Install an npm node package (review required).
Inject a test message into a node.
Send a registered safe command (symbolic mission, not raw velocity).
Stop / emergency-stop a running command.
Create SECRET_REF placeholder.
Redact secrets/hosts/ips before model send.
Destructive/irreversible tools lack confirmation or dry-run support. nodered_deploy_patch, nodered_restart, package_install_node, and local_run_allowed_command can cause significant damage but offer no preview or confirmation step before execution.
Security-sensitive tools (package_install_node, local_run_allowed_command, local_write_allowed_file) lack explicit permission declarations or audit trail guidance. No scope declarations (e.g., 'requires: write:packages', 'requires: execute:shell') visible in definitions.
Error handling is minimal. Tools return success/failure without recovery guidance. E.g., flow_validate_draft returns validation results but does not explain what to do if validation fails or how to fix specific errors.