twwch/workflow-skill/comfyui-workflow

ComfyUI Workflow Generator

Description

Generate ready-to-use ComfyUI workflow JSON files through natural language conversation. Users describe what they want to create, and this skill produces valid ComfyUI API-format JSON that can be directly imported and executed.

Activation

Trigger when the user asks to:

• Create/generate a ComfyUI workflow
• Make an image generation workflow
• Build a video generation pipeline
• Set up txt2img, img2img, img2vid, txt2vid workflows
• Generate a ComfyUI JSON file

Trigger keywords: comfyui, workflow, 工作流, 文生图, 图生图, 文生视频, 图生视频, txt2img, img2img, txt2vid, img2vid, upscale, 放大, inpaint, 重绘, controlnet, lora, sd3, flux, ltxv, mochi, cosmos, stable audio, 音频生成, hunyuan3d, 3d生成, stable cascade, 图生3d

Instructions

Step 1: Understand User Intent

Ask the user what they want to create. Gather:

1. Task type: txt2img / img2img / txt2vid / img2vid / upscale / inpaint / audio / 3d
2. Model: SD1.5 / SDXL / FLUX / SD3 / Wan2.2 / HunyuanVideo / LTXV / Mochi / Cosmos / StableAudio / Hunyuan3D / StableCascade
3. Key parameters: resolution, steps, CFG, sampler
4. Optional modules: ControlNet, LoRA, IPAdapter, upscale model
5. Prompt content: positive and negative prompts

If the user is vague, suggest the most common configuration for their task type.

Step 2: Select and Compose Template

Based on user intent, load the appropriate base template from templates/ directory and customize it:

1. Read the matching template JSON file
2. Modify parameters according to user requirements
3. Add optional modules (ControlNet, LoRA, etc.) by inserting additional nodes and links
4. Validate the workflow DAG structure

Fallback — Official Template Repository: If no matching template is found in the local templates/ directory, search the ComfyUI official template repository: https://github.com/Comfy-Org/workflow_templates/tree/main/templates . The official repo contains 443 templates across 9 categories (Image / Video / Audio / 3D / LLM / Utility / Use Cases / Getting Started / Node Basics). First read templates/index.json to find a matching template name, then download the corresponding JSON file to use as the base template. The official templates use the same format as this project (LiteGraph UI format, version 0.4).

Step 3: Generate and Deliver

1. Output the complete workflow JSON (Litegraph UI format)
2. Save final .json file to current working directory. Any intermediate/temp files go to /tmp/comfyui-workflow/.
3. MANDATORY: Add `models` array to the JSON top level — extract ALL model filenames from nodes (ckpt_name, unet_name, vae_name, clip_name, lora_name, etc.), look up their download URLs from the Model Download Guide below, and include them. This makes ComfyUI auto-prompt download for missing models on import.
4. Provide a brief explanation of the workflow structure
5. List any required custom nodes the user needs to install

Workflow JSON Format Rules

CRITICAL — follow these rules exactly:

• Use Litegraph UI format (NOT API format) — with nodes array, links array, version: 0.4
• Top-level MUST have: id (UUID string), revision (0), last_node_id, last_link_id, nodes, links, groups ([]), config ({}), extra ({}), version (0.4)
• Each node has: id (int), type, pos ([x,y]), size ([w,h]), flags ({}), order (int), mode (0), inputs, outputs, properties, widgets_values
• Node inputs include BOTH connection inputs ({"name": "model", "type": "MODEL", "link": 4}) AND widget inputs ({"name": "seed", "type": "INT", "widget": {"name": "seed"}, "link": null})
• All inputs/outputs should have label and localized_name fields matching name
• Links format: [link_id, source_node_id, source_slot, target_node_id, target_slot, "TYPE"]
• widgets_values array order must match the widget inputs order
• Every workflow MUST have at least one output node (SaveImage, PreviewImage, SaveVideo, etc.)
• All required inputs must be provided (check references/nodes/index.md and the category files under references/nodes/, plus references/node-registry-additions.md)
• Values must satisfy min/max constraints
• For model/file selection inputs (ckpt_name, lora_name, etc.), use the real model filenames from the Model Download Guide below
• MANDATORY: Every workflow JSON MUST include a top-level `models` array. Scan all nodes for model file references (ckpt_name, unet_name, vae_name, clip_name, clip_name1, clip_name2, clip_name3, control_net_name, lora_name, model_name) and add each referenced model to the models array with name, url (direct HuggingFace resolve link), and directory. This enables ComfyUI to auto-detect missing models and prompt the user to download them on import. Example:

"models": [
  {"name": "flux1-dev.safetensors", "url": "https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors", "directory": "diffusion_models"},
  {"name": "clip_l.safetensors", "url": "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors", "directory": "text_encoders"}
]

Common Output Slot Mappings

CheckpointLoaderSimple: 0=MODEL, 1=CLIP, 2=VAE
UNETLoader:             0=MODEL
CLIPLoader:             0=CLIP
DualCLIPLoader:         0=CLIP
VAELoader:              0=VAE
LoraLoader:             0=MODEL, 1=CLIP
CLIPTextEncode:         0=CONDITIONING
EmptyLatentImage:       0=LATENT
KSampler:               0=LATENT
VAEDecode:              0=IMAGE
LoadImage:              0=IMAGE, 1=MASK
ControlNetLoader:       0=CONTROL_NET
UpscaleModelLoader:     0=UPSCALE_MODEL
ImageUpscaleWithModel:  0=IMAGE
CLIPVisionLoader:       0=CLIP_VISION
CLIPVisionEncode:       0=CLIP_VISION_OUTPUT

LLM / Text Generation Nodes (comfyui_LLM_party)

ComfyUI integrates LLM via comfyui_LLM_party custom node package (需安装: https://github.com/heshengtao/comfyui_LLM_party).

IMPORTANT: When user requests a workflow with LLM (script generation, character parsing, storyboard breakdown, prompt enhancement), you MUST:

1. Ask the user which LLM provider they want to use (OpenAI / Claude / Gemini / Ollama / DeepSeek / other)
2. Ask for their model preference or recommend the latest best model for their use case
3. Use the corresponding loader node and configure accordingly

Supported Providers & Loader Nodes

CRITICAL: Exact class_type Names (use these EXACTLY in workflow JSON)

DO NOT invent node names. Only use these exact class_type strings:

Core LLM Nodes — Detailed Specs

LLM_api_loader (class_type: LLM_api_loader)

• Inputs: model_name (STRING), base_url (STRING, optional), api_key (STRING, optional), is_ollama (BOOLEAN, optional)
• Outputs: model (CUSTOM) [slot 0]
• Widget values order: [model_name, base_url, api_key, is_ollama]

LLM (class_type: LLM)

• Inputs (required): system_prompt (STRING, widget), user_prompt (STRING, widget), model (CUSTOM, connection from loader), temperature (FLOAT, widget, default=0.7), is_memory (COMBO "enable"/"disable", widget), is_tools_in_sys_prompt (COMBO, widget), is_locked (COMBO, widget), main_brain (COMBO, widget), max_length (INT, widget, default=1920)
• Inputs (optional): system_prompt_input (STRING, connection), user_prompt_input (STRING, connection), tools (STRING), file_content (STRING), images (IMAGE)
• Outputs: assistant_response (STRING) [slot 0], history (STRING) [slot 1], tool (STRING) [slot 2], image (IMAGE) [slot 3]

mini_party (class_type: mini_party)

• Inputs: input_str (STRING, connection), prompt (STRING, widget), model_name (STRING, widget), base_url (STRING, widget), api_key (STRING, widget), is_enable (BOOLEAN, widget)
• Outputs: STRING [slot 0]

LLM Output → Image/Video Bridge Nodes

CLIPTextEncode_party (class_type: CLIPTextEncode_party)

• Inputs: clip (CLIP, connection), text (STRING, connection from LLM output)
• Outputs: CONDITIONING [slot 0]
• Usage: Bridge LLM text → image generation. Connect LLM.assistant_response → CLIPTextEncode_party.text

json_extractor (class_type: json_extractor)

• Inputs: input (STRING, connection)
• Outputs: STRING [slot 0] (cleaned/repaired JSON)

json_get_value (class_type: json_get_value)

• Inputs: text (STRING, connection), key (STRING, widget)
• Outputs: ANY [slot 0] (extracted value)

LLM Workflow Patterns

Pattern: Multi-stage video production pipeline

LLM_api_loader(model, base_url, api_key) → model
  ↓
LLM(model, system="You are a screenwriter", user="Write a script about: {input}") → script
  ↓
LLM(model, system="Extract all characters as JSON", user=script) → characters_json
  ↓
json_extractor(characters_json) → clean JSON → json_get_value(key="character_1.appearance") → prompt
  ↓
CLIPTextEncode_party(prompt) or CLIPTextEncodeFlux(prompt) → CONDITIONING → FLUX → character_ref_image
  ↓
LLM(model, system="Break into 10 shots with video prompts", user=script) → shots_json
  ↓
For each shot:
  json_get_value(shots_json, "shot_N.description") → scene_prompt
  FLUX(scene_prompt + IP-Adapter(character_ref)) → shot_ref_image
  WanImageToVideo(shot_ref_image + video_prompt) → 10s video clip

Pattern: Simple prompt enhancement

mini_party(input="a cat", prompt="Enhance this into a detailed image prompt") → enhanced_prompt
  ↓
CLIPTextEncode_party(enhanced_prompt) → CONDITIONING → KSampler → image

Node Reference

IMPORTANT: Do NOT read the entire node registry at once. Read only the category you need.