nvidia-tao/tao-skill-bank/tao-finetune-cosmos-reason

Cosmos3 TAO training

Keep one shared model-facing frontend. Backend image fields and contract paths live under backend_contracts in references/skill_info.yaml; image literals are stamped from versions.yaml, while referenced backend YAMLs define native runtime schemas. Never translate between them.

Mandatory runtime intake

Before planning training, collect all of the following. Do not infer a path from history, another user, a prior job, an image, or a developer checkout.

• base_model_path_or_uri. For a Hugging Face model ID or URL, accept an

optional friendly base_model_revision such as a branch or tag. If omitted, resolve main; do not ask the user for a commit SHA. Resolve the selected ref read-only through the Hub API to its immutable commit and seal the model ID, requested ref, and resolved SHA in the plan. A complete local snapshot needs no revision and is sealed by its file fingerprints.

• For Cosmos3-Nano, an explicit input-checkpoint model_type: qwen3_vl

or cosmos3_omni. If the user did not supply it, ask once before planning; never infer the choice from config.json, a model ID, a path name, or a previous run. Explain the two choices in plain language: qwen3_vl uses a compatible Hugging Face checkpoint directly, while cosmos3_omni requires an immutable conversion to exact Qwen3-VL safetensors before training. Record the answer as base_model_format. Cosmos3-Edge is inferred as cosmos3_edge from the resolved model ID and does not present this Nano-only choice.

• Do not expose Omni preparation implementation fields during normal intake.

For Nano, use Qwen/Qwen3-VL-8B-Instruct, resolve both Hub models to immutable commits, and run the selected backend's packaged converter. Framework uses cosmos_framework.scripts.convert_model_to_vlm_safetensors; Cosmos-RL uses cosmos_rl.model_preparation.vlm_safetensors with its pinned isolated converter runtime. Validate an explicitly supplied prepared_checkpoint_path or donor, but never ask for one by default.

• Accept hf_model://nvidia/Cosmos3-Nano directly. If a gated/private model

cannot be resolved, ask the user only to set HF_TOKEN in the session environment; never ask them to discover a SHA or provide the token value in chat.

• explicit video sampling mode: either uniform nframes or fps. FPS mode

may also set min_frames and max_frames; both modes may set clip-time, resize, and pixel-budget fields supported by the selected backend.

• training/validation annotation paths and media roots for conversation-style

or task-aware video supervision, plus optional task selection.

• explicit backend for a comparison; cosmos-framework or cosmos-rl.
• training_mode; dense or peft. PEFT also requires rank, alpha, dropout,

target modules, bias, RS-LoRA, modules-to-save, and adapter precision.

• user-owned results_dir, checkpoint_dir, cache_dir, and, for SLURM,

sqsh_cache_dir, ssh_key_path, mounts, and scheduler settings.

• Runtime order: compute-readable sqsh_path, explicit image, then the selected

backend image in references/skill_info.yaml. On SLURM reuse or convert it once under sqsh_cache_dir. Never compare an SQSH filename with an image tag or request source provenance/SHA.

• Repository paths, commits/trees, branch, base image, build context, and

timestamp are advanced inputs required only for explicit source-build. See references/cosmos-backend-operations.md; never infer a build from runtime selection.

The planner preserves each original path and reports an accessible realpath. Missing required paths fail. A missing supplied SQSH fails; an omitted SQSH selects the packaged image. No historical fallback path or image is allowed.

Nano checkpoint model-type choice

Treat the selected base_model_format as a user decision and verify that it matches the supplied local checkpoint's config.json.model_type. A mismatch fails; it is not permission to relabel or rewrite the source checkpoint.

For qwen3_vl, fingerprint and use a complete compatible Hugging Face Nano checkpoint directly. For cosmos3_omni, tell the user in the launch review that Cosmos-RL will prepare its compatible checkpoint automatically, name the planned output path, and preserve the source checkpoint unchanged. Do not ask the user for an architecture donor, preparation image, or preparation SQSH. The backend-owned preparation step emits a verified qwen3_vl checkpoint under the selected platform's user-owned checkpoint_dir:

• Docker: the local Docker host's checkpoint_dir.
• SLURM: the compute-node-verified shared checkpoint_dir, covered by an

explicit container mount. Run conversion through the SLURM/Pyxis contract; do not write the converted checkpoint to controller-local storage.

Use the packaged Nano architecture mapping and selected backend runtime unless an advanced override was explicitly supplied. Fingerprint the source, architecture mapping, converted config/tokenizer/processor/index/shards, and conversion provenance. Before conversion, bind the sealed plan's backend-native training fields and VLM_SAFETENSORS_PATH to the planned converted checkpoint inside the selected container. Validate that exact output before training without mutating the sealed plan. The original Omni/Hugging Face path remains provenance only and must never remain as the runtime model path. Reuse is allowed only when the exact target has complete matching conversion provenance and passes the same validation.

Public Cosmos3-Edge checkpoint contract

Accept the public model at its resolved immutable revision or a local snapshot; never request a second checkpoint. Apply the model-aware runtime defaults from references/skill_info.yaml and references/cosmos-framework-backend.yaml, preserving separate model and processor-profile fingerprints and each default/override origin.

Backend selection

Run scripts/cosmos_workflow.py resolve first.

An explicit supported backend wins, so users can select Cosmos Framework for Nano training without changing model ownership. Comparative runs reject auto, so both sides of an experiment are deliberately forced. Framework-trained checkpoints use the native exact-key exporter, then the repository-backed TAO evaluation adapter. That does not make Framework a Cosmos-RL version.

Evaluation intake and inheritance

Run scripts/evaluation_workflow.py for every evaluate action and follow references/cosmos-reason-evaluate.md completely. A parent training plan owns all inheritable model, dataset, prompt, preprocessing, precision, checkpoint, and scoring fields; never ask the user to repeat them. Ask once only for the helper's required_user_inputs, execute its automated_actions, and launch only a checksum-valid ready=true plan. Cosmos-RL policy checkpoints require the emitted cosmos_rl_checkpoint_pre_action; Framework DCP inputs require their emitted export pre-action. The ready plan includes a validated spec_bundle.execution; pass it unchanged to the selected platform. Cosmos owns runtime attestation and evaluator configuration; the platform owns launch.

Framework checkpoint pre-action

Before Framework evaluate, inference, or microservice actions, run scripts/framework_checkpoint_action.py plan and its emitted prepare and verify steps. Follow references/cosmos-backend-operations.md; never ask the user to export DCP manually. On SLURM stage only the helper dependency set declared by workflow_contract.action_helper_dependencies; the platform verifies the closed bundle. Use only the verified action_model_path, reuse only matching complete exports, and record the pre-action, manifest, fingerprints, and independent child result.

Required gates

Execute these stages in order and persist their outputs.