zju3dv/habitat-gs/habitat-gs-train

habitat-gs-train

Train and evaluate navigation policies on photo-realistic 3D Gaussian Splatting scenes in habitat-gs. Five tasks are supported, each with a one-click generate → train → evaluate pipeline driven by scripts under scripts_gs/.

IMPORTANT — run from the repo root. Every command below assumes the current directory is the habitat-gs/ project root and that the right conda env is active. The scripts cd to the project root themselves, but paths like data/scene_datasets/gs_scenes/... are repo-relative. Never invent flags — the exact flags each script accepts are in references/.

Pick the task first

RL tasks (PointNav / ImageNav / ObjectNav) are the lightweight, fast path — they run in the base habitat-gs env and need no external repo. Start here for "quickly train and evaluate a navigation policy."

VLN tasks (StreamVLN / Uni-NaVid) are heavy: each needs ≥80 GB VRAM/GPU (StreamVLN has a 24 GB LoRA mode), a separate cloned conda env, and an external repo cloned as a sibling of habitat-gs/. Only go here when the user explicitly wants instruction-following VLN.

The universal flow (all tasks)

1. Prerequisites — env + Habitat-Lab + GS data. Read references/prerequisites.md and verify before anything else. Skipping this is the #1 cause of failures.
2. Generate episodes / trajectories — produces the dataset the policy trains/evals on. The released dataset already ships episodes, so this is OPTIONAL for the standard scenes (the train/eval scripts abort with a clear message if the data is missing).
3. Train — bash scripts_gs/train_<task>.sh --output <dir> [options].
4. Evaluate — bash scripts_gs/eval_<task>.sh --ckpt <ckpt> [options]; read where metrics land in references/outputs-and-metrics.md.

Quick start (PointNav, the simplest task)

conda activate habitat-gs
 
# (optional) generate episodes — skip if the released dataset is present
python scripts_gs/generate_pointnav_episodes.py
 
# train (default 5e8 steps; --output is required)
bash scripts_gs/train_pointnav.sh --output output/pointnav
 
# evaluate a checkpoint
bash scripts_gs/eval_pointnav.sh --ckpt output/pointnav/checkpoints/ckpt.0.pth

ImageNav and ObjectNav are identical in shape — swap pointnav for imagenav / objectnav (their default step counts and episode generators differ; see their references).

How to use this skill

1. Confirm the task with the table above. If the user just says "train a nav policy", default to PointNav (fastest to a working result) and say so.
2. Check prerequisites (references/prerequisites.md): correct conda env, Habitat-Lab installed with the numpy pin patched, GS data under data/scene_datasets/gs_scenes/, and for ObjectNav generation the SAM + CLIP checkpoints.
3. Decide whether to generate data. If data/scene_datasets/gs_scenes/episodes/<task>/{train,val}/content already exists, skip generation. Otherwise run the generator (read the task reference — ObjectNav needs extra models; VLN needs a VLM endpoint).
4. Launch training with train_<task>.sh. Training is long-running — launch it in the background and poll, or hand the user the exact command. For multi-GPU pass --num-gpus N --num-envs M. See references/training-and-finetuning.md for resume vs. fine-tune vs. encoder-transfer.
5. Evaluate with eval_<task>.sh --ckpt <checkpoint.pth>. Prefer a single checkpoint file — the directory form is a polling watcher that can hang (see references/task-pointnav.md). Add --video-dir DIR (RL) / --save-video (VLN) to record rollouts. For a quick check, append habitat_baselines.test_episode_count=3.
6. Report metrics by reading them from the eval output (RL: SPL / Success / DistanceToGoal printed to stdout & eval.stdout; VLN: SR / SPL / OSR / DTG, Uni-NaVid also writes summary.json). See references/outputs-and-metrics.md.

Reference index

Smoke test the whole pipeline

scripts_gs/_verify_full_pipeline.sh runs train+eval for all five tasks on a tiny 3-train / 3-val subset. It is the authoritative end-to-end command reference, but it hardcodes a foreign conda path and work dir — edit those before running. Details in references/troubleshooting.md.

Troubleshooting (quick)