Loading ToolBench
Drive a remote ComfyUI instance: inspect models/nodes, queue API-format workflows, retrieve rendered images locally.
Server has 9 well-named tools with clear verb-noun patterns (status, list_models, run_workflow, etc.). All tools have descriptions (10 - 200 chars, within baseline 10 - 1024 range). Input schemas are present and typed for all tools. However, output schemas are undocumented, tools return JSON strings without declaring field structure. Parameter descriptions are minimal (e.g., 'Model folder to list' lacks format/constraint details). Error handling is present but generic (e.g., 'rejected by ComfyUI' without recovery guidance). No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels. Missing idempotency guidance for retry-heavy operations like run_workflow.
Drop all pending jobs from the queue.
Fetch the result of a previously queued prompt, downloading its images locally.
Cancel the currently running job.
List model files on the pod. Omit `folder` to list available folders (checkpoints, loras, vae, unet, clip, controlnet, ...).
List installed node class names, optionally filtered by a case-insensitive substring.
Full input/output schema for one node class. Use before wiring it into a workflow.
Queue a ComfyUI API-format workflow and wait for it to finish. `workflow` is the JSON from ComfyUI's "Export (API)" — a dict keyed by node id, each value having "class_type" and "inputs". Returns the prompt_id plus any output images, which are downloaded locally (see `local_path`) so they can be read.
Output schemas undocumented. Tools return JSON strings (e.g., run_workflow returns {prompt_id, status, images, error}) but LLMs cannot parse structure without explicit schema. Agents cannot reliably extract fields for downstream calls.
Tool annotations missing. Tools like clear_queue (DESTRUCTIVE) and interrupt (REVERSIBLE) lack readOnlyHint/destructiveHint/idempotentHint. LLMs cannot determine which tools are safe to retry or require confirmation.
Parameter descriptions lack constraints. 'folder' in list_models says 'Model folder to list' but does not specify valid folder names (checkpoints, loras, vae, etc.). 'timeout_s' in run_workflow lacks min/max bounds. LLMs cannot validate inputs.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | <=2025-11-25 | v2 |
ComfyUI version, VRAM/RAM, and current queue depth.
Upload a local image into ComfyUI's input folder so workflows (img2img, controlnet, inpaint) can reference it by the returned name.
Error handling lacks recovery guidance. run_workflow returns {error: 'rejected by ComfyUI', detail: ...} but does not tell LLM what to do next. get_result returns 'pending_or_unknown' without suggesting retry timing or alternative actions.
Idempotency not documented. run_workflow queues a job and waits; if the agent retries due to timeout, a duplicate workflow may be queued. No guidance on idempotent retry behavior or deduplication.