timmyy123/llm-hub/train-lora

Train LoRA Skill

Use this workflow to validate Draw Things LoRA training end to end with draw-things-cli.

Goal

Train on a tiny local dataset, watch loss and scaler health, then verify visually with a reference/base/LoRA comparison.

Build Once

Build the optimized CLI first:

bazel build --compilation_mode=opt //Apps:DrawThingsCLI

Use the built binary for every run:

bazel-bin/Apps/DrawThingsCLI

Do not switch between bazel run and bazel-bin/... during one validation cycle unless you need to. Reuse the same binary so compile/runtime behavior stays comparable and permission prompts stay predictable.

Runtime Notes

• Graph compile can be quiet for a long time. With --compilation_mode=opt, 5 to 30 minutes is possible on heavy trainers. Do not assume a hang too early.
• Training checkpoints are written into the models directory.
• If the environment requires command approvals, ask once for a stable command shape and stable output/log names, then rename artifacts afterward.
• For local unregistered LoRAs, prefer passing explicit loras[].version during generation instead of depending on custom_lora.json.

Dataset Setup

For a single-image reconstruction check:

• Create a local dataset directory.
• Put the image in that directory.
• Add a matching .txt caption file beside it.
• Keep the caption minimal for trigger-only tests, for example:

zimgdogref

Validation Ladder

Use this order:

1. Run a 1-step smoke test to confirm the graph compiles, the loss is finite, and a checkpoint is written.
2. Run a 20-step probe to confirm scale stays healthy and loss is not obviously blowing up.
3. Run a 100-step check to see whether like-for-like timestep bands decline.
4. Run a 500-step run before declaring the trainer healthy.
5. If you are validating a new attention backend, run the 500-step check on that intended backend, not only on a fallback path.
6. Generate a base image and a LoRA image with the same prompt, seed, and settings.
7. Compose reference/base/LoRA into one image for visual review.

Baseline Train Command

Use this as the generic 512x512 single-image baseline:

bazel-bin/Apps/DrawThingsCLI train lora \
  --models-dir /Users/liu/Library/Containers/com.liuliu.draw-things/Data/Documents/Models \
  --model MODEL.ckpt \
  --dataset /tmp/single_image_dataset \
  --output RUN_NAME \
  --name RUN_NAME \
  --steps 500 \
  --rank 32 \
  --scale 1 \
  --learning-rate 0:4e-4 \
  --gradient-accumulation 4 \
  --warmup-steps 20 \
  --save-every 100 \
  --width 512 \
  --height 512 \
  --seed 7 \
  --config-json '{"steps_between_restarts":200}' \
  --no-download-missing \
  --offline

Model Baselines

Scaler Rules

• Healthy scale is architecture-dependent; choose it from the model's numeric contract.
• Do not set scale lower than 1 to make a run stable. That hides overflow and can prevent useful learning.
• If the model does not apply internal scaling that shrinks gradients, start from 32768.0.
• If the model has explicit internal downscaling or projection compensation, use the validated lower scale for that model family and record why.

FLUX.1

• Base model: flux_1_dev_q8p.ckpt
• Validated guidance settings:
• guidanceScale = 3.5
• guidanceEmbed = 3.5
• shift = 2
• resolutionDependentShift = false
• Healthy scale is typically 32768.0

Z Image Turbo

• Base model: z_image_turbo_1.0_i8x.ckpt
• The validated training baseline is the generic command above.
• The validated trainer scale is 1024.0
• For generation validation, use cfg = 1

Z Image Base

• Base model: z_image_1.0_q8p.ckpt
• The validated training baseline is the generic command above.
• The validated trainer scale is 1024.0
• For generation validation, keep the model’s recommended Base path:

{"sampler":17,"shift":1.8776105999999999,"resolutionDependentShift":true}

• A validated comparison used cfg = 4

Qwen Image BF16

• Base model: qwen_image_2512_bf16_i8x.ckpt
• The validated training baseline is the generic command above.
• Healthy scale is 32768.0
• Use the exact training caption first before trying richer prompts

What To Watch During Training

• Raw loss is noisy because each step samples a different timestep. Do not expect monotonic decline step by step.
• Compare like-for-like timestep bands instead.
• Mid/high timestep bands should usually improve first.
• Low timestep spikes can happen, but they should stay bounded.
• For flow-style objectives, low timestep loss is not always the easiest band. If the target includes a full noise or velocity term that is weakly visible in the low-timestep input, low timestep bins can be intrinsically harder.
• If scale steadily collapses, something is seriously wrong.
• If scale collapses only on a new backend, compare against the known-stable backend before changing learning rate or dataset settings.
• If scale collapses on a model with rotary applied through cmul, check whether trainer rotary constants are expanded to the real query/key head count.
• Before blaming the optimizer, confirm the checkpoint is real:
• nontrivial file size
• lora_up tensors are not all zero

Generation Validation

Always compare base and LoRA with the exact same:

• prompt
• seed
• width
• height
• steps
• CFG
• model-specific sampler/shift settings

Use the exact training caption first. If that fails, richer prompts are not useful for debugging.

For non-distilled base models, do not under-sample the generation validation. Use the model's real baseline settings, including enough steps, the correct CFG behavior, and the correct sampler family.

For local LoRAs, pass explicit version metadata:

"loras": [
  {
    "file": "RUN_NAME_500_lora_f32.ckpt",
    "version": "MODEL_VERSION",
    "weight": 1.0
  }
]

If the model also needs LoRA mode metadata, pass mode too.

Example Generate Commands

Z Image Turbo

bazel-bin/Apps/DrawThingsCLI generate \
  --models-dir /Users/liu/Library/Containers/com.liuliu.draw-things/Data/Documents/Models \
  --model z_image_turbo_1.0_i8x.ckpt \
  --prompt zimgdogref \
  --width 512 \
  --height 512 \
  --steps 15 \
  --cfg 1 \
  --seed 7 \
  --config-json '{"loras":[{"file":"RUN_NAME_500_lora_f32.ckpt","version":"z_image","weight":1.0}]}' \
  --offline \
  --no-download-missing \
  --output /tmp/zimg_lora.png

Z Image Base

bazel-bin/Apps/DrawThingsCLI generate \
  --models-dir /Users/liu/Library/Containers/com.liuliu.draw-things/Data/Documents/Models \
  --model z_image_1.0_q8p.ckpt \
  --prompt zimgdogref \
  --width 512 \
  --height 512 \
  --steps 20 \
  --cfg 4 \
  --seed 7 \
  --config-json '{"sampler":17,"shift":1.8776105999999999,"resolutionDependentShift":true,"loras":[{"file":"RUN_NAME_500_lora_f32.ckpt","version":"z_image","weight":1.0}]}' \
  --offline \
  --no-download-missing \
  --output /tmp/zimg_base_lora.png

Compose A Review Image

Use ffmpeg to compose reference, base, and LoRA side by side:

ffmpeg -y \
  -i /tmp/single_image_dataset/dog.png \
  -i /tmp/base.png \
  -i /tmp/lora.png \
  -filter_complex hstack=inputs=3 \
  -frames:v 1 \
  /tmp/compare.png

Expected Outcomes

• Base should usually look generic or unrelated to the exact training identity.
• A healthy LoRA should pull noticeably toward the training subject by 100 to 500 steps.
• For single-image dog tests, the correct check is not “perfect reconstruction”; it is whether the LoRA image is materially closer to the reference than the base image.