slavasexton/comfyui-agent-kit/comfyui

ComfyUI: driving the local install

Use this whenever the task involves generating or rendering images, video, or audio with ComfyUI, or building/running a ComfyUI workflow. Read it first, then act.

Files in this kit (pull the right one on demand)

Only this SKILL.md auto-loads; everything else is read when relevant, so route to it instead of leaving it unread. Two layouts, same files: in the INSTALLED skill everything sits flat next to this file, so docs/TASKS.md below means TASKS.md here and docs/NODE_LIBRARY/ocio.md means NODE_LIBRARY/ocio.md; likewise a script the docs name under shared/tools/ in the repo sits in tools/ next to this file once installed. In the repo those prefixes are literal. If a path does not resolve, drop the docs/ and look next to this file before concluding the file is missing.

• `MODELS.md` (next to this file) - the INDEX of per-model prompt recipes. Look the model up in its table, then read that family file under MODELS/ BEFORE writing the prompt. Two reads, not one: the index does not carry the recipes.
• the sibling `minimax-h3` skill (invoke it by name; on disk it sits beside this skill, ../minimax-h3/ on Claude Code and Codex, minimax-h3/ on Gemini and Qwen) - the dedicated MiniMax H3 (Hailuo 3) skill: prompt format (the three named fields, <d> dialogue, camera vocabulary), reference labelling, quants and acceleration, and a symptom-to-cause table. Read it for ANY H3 prompt or local-weights question; MODELS.md keeps the node-level detail.
• the sibling `krea` skill (invoke it by name; beside this skill on disk) - the dedicated Krea skill: the fork between Krea's hosted API (Krea2ImageNode / Krea2StyleReferenceNode, per-image pricing, moodboards, capped at 1K) and its open weights, the FLUX.1 Krea Dev graph, Krea Realtime 14B and why its only ComfyUI pack is an unproven lead, and the Krea 2 custom-node packs for ControlNet / identity editing / conditioning control. Read it for the API path or the model choice; MODELS.md keeps the local Krea 2 graph.
• the sibling `seedance` skill (invoke it by name; beside this skill on disk) - the dedicated ByteDance Seedance skill: the three task types and the word that switches between them, the @Image 1 label syntax, the full-width symbol set, shot sequencing, the asset-count rule, the timing rules that reversed in 2.5, and a failure table. Read it for ANY Seedance prompt; MODELS.md keeps the node-level detail and the price maths.
• `docs/TASKS.md` - a named common job (generate image / video / audio / 3D, upscale, remove background): the local end-to-end flow for that task, a shortcut layer over this manual.
• `docs/NODE_LIBRARY/smart-upscaler.md` - our Smart Upscaler pack (11 nodes): tiled upscaling that writes a separate verified prompt per tile. Read it when a tiled upscale of a BUSY or MIXED scene keeps producing confidently wrong tiles or disagreeing seams; the cheaper sampler-tilers in ADVANCED.md stay the right call for uniform subjects.
• `docs/MODEL_INDEX.md` - the full classified list of all 160 models (recipe / utility / template-only); check whether a named model has a recipe, is a utility, or is template-only.
• `docs/NODE_LIBRARY/training.md` - the nodes that let a graph MAKE a model, not just prompt one: core's

TrainLoraNode / SaveLoRA / LossGraphNode plus the 16 dataset nodes (MakeTrainingDataset, ResolutionBucket, the image-text loaders, video temporal crops) and the full chain wired end to end. Read it whenever someone asks to train or fine-tune anything; docs/TASKS.md has the short route.

• `docs/ADVANCED.md` - hard tasks: real strengths, gotchas + workarounds, temporal stability, high-detail matting, crop-and-stitch inpaint, PBR, and the verified tool table with licenses.
• `docs/KNOWN_ISSUES.md` - read BEFORE building, so you do not wire around a currently-broken path.
• `docs/NODE_LIBRARY/_INDEX.md` - the per-node reference (Nuke-style): for any node, what each input / output is for, how it behaves, bugs + fixes, anti-patterns, and where it slots in a graph. Start here for ANY node question, then query get_node_info for live I/O. When you use or meet a node not in it, add the entry before finishing (docs/NODE_LIBRARY/_SCHEMA.md).
• `workflow_layout.py` - before saving ANY workflow you build, arrange and verify it IN CODE: auto_layout(wf) positions nodes left-to-right by dependency depth with parallel branches stacked and ZERO overlaps; inspect(wf) reports overlaps / crossings / bounds from the coordinates; fit_group(wf, title) wraps the laid-out nodes in a backdrop that FULLY covers the functional group (edge to edge, none sticking out). NEVER judge a graph's layout from a screenshot (it burns tokens, and clients hit the same wall) - read the positions.
• `docs/NODE_LIBRARY/ocio.md` - our own ComfyUI-OCIO pack, v1.3.0 (eleven Nuke-style OpenColorIO nodes: Read / Write / Player, six color operators, and the OCIO VAE Decode / VAE Encode pair that decodes without the stock 0..1 clamp; published, github.com/SlavaSexton/ComfyUI-OCIO). Read it for ANY color-management / VFX color task (load a sequence, grade in ACES, write ProRes / EXR, keep values above 1.0 and below 0 out of a generative model). v1.3.0 renamed OCIO Write's `from_colorspace` to `input_colorspace`, and an API-format graph carrying the old key is refused, so read the file before building one.
• `docs/BUILDING_NODES.md` - the hard-won field guide to WRITING a custom node pack (widget order, the combo-validation trap, the JS front-end, server routes, ComfyUI facts, verify-on-real-files). Read it first when you write or modify a custom node, alongside the comfyui-node-* skills.
• `docs/KIJAI.md` - the kijai ecosystem (his ComfyUI wrappers and nodes: Wan / Hunyuan / CogVideoX / Florence2 / KJNodes / SUPIR / FramePack / SAM2 / FluxTrainer / IC-Light / DepthAnythingV2 and ~50 more) - what each does + node I/O, what is active vs legacy by date, and the supersede map (old -> better). Read it for ANY kijai tool, and to pick the current option over a sunset wrapper.
• `docs/NODE_LIBRARY/radiance.md` - reverse-engineered reference for `fxtdstudios/radiance` (a pro 32-bit color-science / HDR / VFX suite, 78 nodes + a v3 rewrite; the strongest public pack in our OCIO / color domain). Read it for a color-science / HDR / VFX-viewer reference implementation, when improving OUR ComfyUI-OCIO pack (it carries the ranked "what to steal" list: processor caching, WebGL 32-bit viewer, OpenEXR writer, LogC3-EI / LogC4), or when a task touches radiance nodes. Its node-building lessons are folded into BUILDING_NODES.md.
• `docs/LTX2_TRAINING.md` - when the user works with LTX-2 and wants behavior a LoRA captures, offer to train one (official Lightricks trainer).
• `docs/EXAMPLE_WORKFLOWS.md` - worked end-to-end examples + the multi-model image-edit shootout.
• `docs/NODES.md` the in-graph Claude nodes (billing / purpose); `docs/LAYERS.md` the four install layers; `docs/BOOTSTRAP.md` first-run machine setup; `docs/AGENTS.md` per-agent matrix (Claude / Codex / Gemini / Qwen); `docs/UPDATING.md` the weekly model + bug update loop.

Your machine

Read `machine.md` next to this file before the first ComfyUI call of a session. It carries this install's real paths, GPUs, template-library location and launch command. It is deliberately NOT part of this file: the installer overwrites SKILL.md on every update, so a machine block living here was destroyed by the next git pull plus reinstall, and the bootstrap had to be redone (2026-08-06 audit). machine.md is written once and never overwritten.

If machine.md is missing or still full of angle brackets, run the bootstrap now (docs/BOOTSTRAP.md): health_check, or comfy_client.alive() plus GET /system_stats and /object_info, then write the real values into machine.md. Never assume another machine matches an example.

The four layers (what this kit installs)

1. Knowledge + client: this SKILL.md and comfy_client.py (stdlib, no deps).
2. MCP driver: comfyui-mcp (artokun, MIT): ~90 structured tools so Claude operates ComfyUI directly

(generate, build/edit/validate graphs, model download, queue, VRAM, diagnostics, restart). Prefer its tools over hand-POSTing /prompt when present. It rides the MCP SDK's 1.x line, i.e. the revision BEFORE the stateless 2026-07-28 spec; that stays supported through a twelve-month deprecation window, so there is nothing to change. If you write your OWN MCP server, read the protocol section in docs/LAYERS.md first: an unbounded mcp>=x pin now pulls SDK 2.x, which removed mcp.server.fastmcp and breaks fresh installs.

1. In-graph Claude nodes: Claude as a step INSIDE a workflow (prompt enrichment, vision QA on the output).
2. Node-building skills: comfyui-node-* (V3 API) for when we write or modify a custom node.

docs/LAYERS.md explains each; install.ps1 / install.sh wires them up.

The client (no extra deps, stdlib only)

comfy_client.py lives next to this SKILL.md. Import and use:

import sys; sys.path.insert(0, r"<this skill dir>")
import comfy_client as c
c.alive()                                  # True if the API answers (override host with COMFY_HOST env)
c.run("path/to/workflow_api.json",
      overrides={"6.text": "a cinematic dragon, dark studio light", "3.seed": 12345},
      outdir=r"...\assets")                # queues, waits, downloads -> returns saved file paths

API surface: alive(), run(workflow_path, overrides, outdir, timeout), and the pieces queue(workflow), wait(prompt_id), download_outputs(rec, outdir), apply_overrides(wf, overrides). Override keys are "<nodeId>.<inputName>" (node ids and input names come straight from the workflow JSON). The MCP driver (Layer 2) does the same and more; use it when available, fall back to this client otherwise.

Workflow JSON (API format)

ComfyUI runs the "API format" graph: a dict { "<nodeId>": { "class_type": "...", "inputs": {...} }, ... }. To get one: in ComfyUI enable Settings -> Enable Dev mode Options, build the graph, then Save (API Format). Official starting graphs: Workflow -> Templates browser (per model). Save those as API Format, then parameterize.

To parameterize a graph, read it and find:

• the positive prompt: a CLIPTextEncode node, override .text.
• the seed: a KSampler / sampler node, override .seed (use a varied seed per call; do not hardcode).
• dimensions: an EmptyLatentImage / EmptySD3LatentImage node, override .width / .height.
• steps/cfg/sampler/scheduler: on the sampler node. Keep the template's values unless asked, they are model-tuned.

Template library (the SOURCE OF TRUTH, mix and match)

The official Comfy-Org workflow templates are the source of truth for how to do any task in ComfyUI. The kit clones them (sparse) to a local folder and builds a compact lookup index. Default location set by the installer; record it in the machine block above. Master index: templates/_quick_index.json (name -> title, category, models, tags, mediaType, vram, description), regenerate with shared/tools/gen_quick_index.py. Update: git pull in the clone, then rerun the generator.

Flow: read _quick_index.json, find the template whose name/models/tags match the request, read THAT one templates/<name>.json, parameterize it. New templates use SUBGRAPHS: the real pipeline is inside definitions.subgraphs[0], exposed params (text, width, height, seed, steps, model names) are in subgraphs[0].inputs, traced to inner nodes via the outer node's properties.proxyWidgets. Mix and match the blueprints/ (reusable subgraph bricks: text_to_image_z_image_turbo, image_to_video_ltx_2_3, image_upscale_z_image_turbo, remove_background_birefnet, ...).