junchao-cs/solarwm/solar-training-ops

SolarWM Training and Inference Operations

Operate from a SolarWM source checkout. Treat the current checkout, resolved configuration, active runtime, and generated manifests as authoritative. A configuration filename or run name is descriptive, not proof of the route or runtime state.

Scope

Own these tasks:

• select a supported training or inference route;
• adapt and resolve a checked-in example configuration;
• validate distributed topology and batch arithmetic;
• launch, initialize, resume, monitor, and stop a task-owned run;
• verify checkpoints, periodic validation, standalone inference, and comparison

outputs;

• diagnose non-finite values, OOM, distributed hangs, shape failures, checkpoint

incompatibility, and missing validation artifacts.

Do not use this skill to build or publish datasets, preencode samples, acquire model weights, build containers, or provision a particular cloud or cluster. Do not turn a diagnosis-only request into a restart or code change.

Read only the relevant documentation

Read the document for the selected backend before acting:

• Wan 2.2 TI2V-5B
• Wan 2.2 I2V-A14B
• LTX-2.5
• MiniMax-H3

Read runtime environments for backend setup. Read the data contract only when the request involves indexes, sampling, storage transport, camera inputs, or shard access.

What each training route learns

Stage names describe the role of a training phase; they do not imply that every backend supports the same objective. Use the current route list as the capability boundary.

The training progression is Stage0.5 FM → Stage1 TF-AnyFlow → Stage2 DMD via SGF.

• Stage0.5 establishes the bidirectional full-clip model.
• Stage1 combines causal teacher forcing and the AnyFlow loss in one stage,

producing weights that can initialize Stage2/DMD without a separate ODE or consistency-distillation initialization stage.

• Stage2 DMD via SGF starts a new three-role run. Initialize the student from

Stage1 AnyFlow EMA and the frozen teacher plus trainable critic from compatible Stage0.5 FM weights, exactly as declared in the Stage2 config.

The optional Stage1 TF-FM baseline remains implemented through train_stage1_tf_fm_81f.yaml, but it is not the released Stage2 initializer.

Validate each transition with the next stage's actual load path and deterministic test-index selection before a long run.

Loss terminology:

• FM uses a point on a straight clean/noise path and regresses its velocity.

Wan and LTX use x_sigma=(1-sigma)*x0+sigma*eps with target eps-x0.

• H3 FM uses the equivalent reverse time coordinate,

x_t=t*x0+(1-t)*eps, with target x0-eps.

• AnyFlow predicts a map from noise level t toward a requested endpoint

r; it is not an alias for ordinary FM or merely a different loss weight.

• DMD via SGF trains the student through a teacher/critic-derived gradient

on the student's own rollout, while retaining a separate FM objective for the critic.

Wan I2V-A14B and LTX-2.5 expose no Stage1 or Stage2 training route in the current release. Reject those combinations before model allocation.

For H3 Stage1/2, use the backend guide. Keep the automatically frozen validation plan, encoded-silence condition and tail-camera rules fixed. Stage2 EMA begins at student update 39 (outer step 196); validation at steps 20 and 100 is LIVE only, and step 200 includes LIVE and EMA.

Operating workflow

1. Establish the exact action and route

Identify the model family, stage, objective, frame profile, data encoding, checkpoint mode, and whether the request is training, periodic validation, or standalone inference.

List the routes implemented by the current checkout:

solar config routes

Select the closest file under configs/solar_wm/. Do not transfer stage, objective, latent layout, model assets, or inference settings between backends. If the requested combination is not listed, report it as unsupported instead of allocating a model.

2. Resolve the run before allocating GPUs

Inspect the active environment and the selected data controls: Set the SOLAR_* shell variables below from the intended configuration before running the examples.

solar environment probe
solar data inspect /path/to/index.jsonl.gz
solar data plan /path/to/index.jsonl.gz \
  --seed "${SOLAR_SEED}" \
  --pixel-frames "${SOLAR_PIXEL_FRAMES}" \
  --world-size "${SOLAR_RAW_WORLD_SIZE}" --rank 0
solar config resolve \
  --config configs/solar_wm/<backend>/<config>.yaml \
  --output /path/to/resolved-preview.json

Review the resolved action and route, model assets, frame geometry, data index and transport, camera transform, world and sequence-parallel sizes, effective global batch, initialization or resume mode, EMA, validation passes, output directory, and save/validation intervals.

Supply deployment paths with explicit --set values or a task-local config. Do not commit credentials or deployment-specific absolute paths. Reject a run when unresolved /path/to/... placeholders remain in fields it will use.

The stored camera arrays keep their published convention and scale. A configured camera-translation transform is applied by the model and must agree across training, checkpoint loading, periodic validation, and standalone inference.

3. Check distributed arithmetic

Compute the topology from the resolved values:

logical_dp_world_size = world_size / sequence_parallel_size
global_batch_size = logical_dp_world_size * micro_batch_size * gradient_accumulation

Require divisibility and agreement with the configured global batch. Sequence- parallel peers must consume the same logical occurrence, start frame, prompt, camera, noise, and timestep state. Dataset ownership uses logical data-parallel rank, not raw rank. Every collective must execute in the same order on every member.

Do not assume a topology validated for one backend, resolution, frame count, or attention layout applies to another.

4. Run the smallest representative preflight

Keep preflight proportional to the change. Before a long or multi-node run, use the selected backend's documented launch path with the intended environment, data path, and distributed settings. Start with the shortest representative run that proves the route-specific gates:

• every rank joins and reads the intended logical samples;
• forward, backward, optimizer, scheduler, and EMA behavior are finite;
• a checkpoint transaction completes and can be loaded;
• the configured recipe test-index selection materializes successfully;
• every configured validation pass produces its videos, per-sample manifests,

comparison outputs, and completion marker.

A short data or distributed bootstrap does not replace the model-step and full validation-output checks. Stop at the first topology, compatibility, non-finite, or artifact-count mismatch, fix the cause, and rerun only the affected gate.

5. Launch through the unified entrypoints

Use --standalone only for a single node. A multi-node deployment must supply its own rendezvous coordinates while preserving the resolved world size and rank map.

torchrun --standalone --nproc-per-node="${SOLAR_LOCAL_WORLD_SIZE}" \
  -m solar train \
  --config configs/solar_wm/<backend>/<train-config>.yaml \
  --set distributed.world_size="${SOLAR_RAW_WORLD_SIZE}" \
  --set train.global_batch_size="${SOLAR_GLOBAL_BATCH_SIZE}" \
  --set runtime.output_dir=/path/to/new-output
 
torchrun --standalone --nproc-per-node="${SOLAR_LOCAL_WORLD_SIZE}" \
  -m solar infer \
  --config configs/solar_wm/<backend>/<infer-config>.yaml \
  --set distributed.world_size="${SOLAR_RAW_WORLD_SIZE}" \
  --set runtime.output_dir=/path/to/new-output

Inference sample counts are backend-configured: Wan and H3 use validation.sample_count, while LTX uses inference.sample_count. Override the field declared by the selected inference YAML rather than adding a training batch field to an inference run.

Use a new output directory for a new run. Rank zero writes resolved-config.json, launch-manifest.json, and run-result.json; inspect them instead of reconstructing the effective command from memory.

6. Preserve checkpoint semantics

Follow the selected config's checkpoint block and backend documentation. Distinguish full resume from weight initialization:

• full resume restores the model, optimizer, scheduler, completed step, EMA,

random state, and data-stream state required by that route;

• weight initialization starts a new optimization run and does not inherit the

prior optimizer or step counter;

• role-based routes require every declared role and transaction member;
• an expected checkpoint that is missing, incomplete, or incompatible is an

error, not permission to start fresh.

Validate model family, stage, objective, camera transform, tensor layout, and the weight selection declared by the chosen route before model allocation. The Wan5B Stage2 camera-length inference route accepts the released checkpoint directory as one model and does not expose LIVE/EMA selection. Accept a saved checkpoint only after its completion marker and declared members are durable. Resolve the model role from release-manifest.json and retain that resolved role in output provenance. Decode horizons longer than 60 latents as consecutive cached VAE tiles, clearing the temporal cache only before and after the sequence.

7. Keep inference and validation aligned