jaccen/awesome-gaussian-skills/3dgs-training-debugger

Iter     Loss    PSNR    Gaussians   VRAM     Notes
0        0.45    16.2    1(x SfM)    4.2 GB   Init from SfM points
500      0.22    20.1    12,000      5.1 GB   First ADC cycle
1000     0.15    23.5    45,000      7.8 GB   Rapid growth phase
2000     0.09    26.8    180,000     12.4 GB  Growth slowing
5000     0.05    29.2    350,000     14.2 GB  Near convergence
7000     0.04    30.1    380,000     14.5 GB  Fine-tuning
10000    0.03    30.4    390,000     14.6 GB  Final (shutting densif)
15000    0.03    30.5    395,000     14.6 GB  Densif frozen, opacity fine-tune
30000    0.025   30.6    395,000     14.6 GB  Final model

SYMPTOM: Training crashed or NaN
│
├── NaN/Inf in loss?
│   ├── Check gradient norms → extremely large?
│   │   └── Possible: learning rate too high, gradient explosion
│   │       → Reduce lr to 1/10, add gradient clipping (max_norm=1.0)
│   ├── Check after ADC cycle → NaN appears right after densification?
│   │   └── Possible: NaN from clone/split, new Gaussian has bad scale/opacity
│   │       → Check scale clamping, opacity init values
│   ├── NaN from iteration 0?
│   │   └── Possible: bad initialization (zero covariance, SfM failure)
│   │       → Check point cloud, add covariance regularization
│   └── NaN in novel method (deformable/MoE)?
│       └── See Section 9: Novel Method Stability
│
├── CUDA OOM?
│   ├── During training (non-ADC)?
│   │   ├── Gaussian count reasonable but still OOM?
│   │   │   └── Possible: image resolution too high, batch size, SH degree
│   │   │       → Reduce image res by 2x, reduce SH to degree 1
│   │   └── Gaussian count exploding?
│   │       └── Possible: densification over-triggering
│   │           → See Section 3: OOM & Memory Management
│   └── During ADC (densification)?
│       └── Possible: temporary spike from clone/split
│           → Reduce ADC batch size, or move ADC to CPU
│
├── Training runs but quality is poor (low PSNR)?
│   ├── Gaussian count too low?
│   │   └── Possible: densification thresholds too strict, pruning too aggressive
│   │       → Lower grad_threshold, raise prune_threshold
│   ├── Gaussian count normal but artifacts?
│   │   └── See Section 4: Artifact Diagnosis
│   ├── Convergence stalled early?
│   │   └── See Section 6: Convergence Analysis
│   └── Specific views are bad?
│       └── Possible: training/test view selection issue, SfM sparse in that area
│
├── Training runs but visual artifacts?
│   ├── Floaters (small isolated Gaussians)?
│   │   └── See Pattern FP-01 in references
│   ├── Blur / over-smoothing?
│   │   └── See Pattern FP-02
│   ├── Ghosting / duplicate geometry?
│   │   └── See Pattern FP-03
│   ├── Holes / missing regions?
│   │   └── See Pattern FP-04
│   └── Color bleeding / SH artifacts?
│       └── See Pattern FP-05
│
└── Checkpoint resume gives different results?
    └── See Section 8: Checkpoint & Resume

VRAM_peak ≈ Model_VRAM + Optimizer_VRAM + Raster_VRAM + Gradient_VRAM + ADC_spike
 
Where:
  Model_VRAM   = N_gaussians × bytes_per_gaussian
  Optimizer_VRAM = 2 × Model_VRAM (Adam: momentum + variance)
  Raster_VRAM  = H × W × n_channels × num_images_in_batch × 4 bytes
  Gradient_VRAM = Model_VRAM (gradients for all params)
  ADC_spike    = 1.5 × Model_VRAM (temporary allocation during clone/split)
 
  bytes_per_gaussian ≈ 59 × 4 = 236 bytes
    (3 position + 3 scale + 4 rotation + 1 opacity + 48 SH (degree 3) = 59 floats)

Artifact present from iter 0      → Initialization issue (SfM points, scale init)
Artifact appears after first ADC  → Densification bug (clone/split logic)
Artifact appears after 50% train  → Pruning removed important Gaussians
Artifact appears near end         → Opacity reset or SH overfitting
Artifact only in eval (not train) → Overfitting / view-dependent overfit