Skill v1.0.1
currentAutomated scan100/100name: accelerate description: Run PyTorch training across GPUs with minimal changes. version: 1.0.1 author: Orchestra Research license: MIT dependencies: [accelerate, torch, transformers] platforms: [linux, macos, windows] metadata: hermes: tags: [Distributed Training, HuggingFace, Accelerate, DeepSpeed, FSDP, Mixed Precision, PyTorch, DDP, Unified API, Simple]
HuggingFace Accelerate - Unified Distributed Training
Quick start
Accelerate simplifies distributed training to 4 lines of code.
Installation:
pip install accelerate
Convert PyTorch script (4 lines):
import torch+ from accelerate import Accelerator+ accelerator = Accelerator()model = torch.nn.Transformer()optimizer = torch.optim.Adam(model.parameters())dataloader = torch.utils.data.DataLoader(dataset)+ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)for batch in dataloader:optimizer.zero_grad()loss = model(batch)- loss.backward()+ accelerator.backward(loss)optimizer.step()
Run (single command):
accelerate launch train.py
Common workflows
Workflow 1: From single GPU to multi-GPU
Original script:
# train.pyimport torchmodel = torch.nn.Linear(10, 2).to('cuda')optimizer = torch.optim.Adam(model.parameters())dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)for epoch in range(10):for batch in dataloader:batch = batch.to('cuda')optimizer.zero_grad()loss = model(batch).mean()loss.backward()optimizer.step()
With Accelerate (4 lines added):
# train.pyimport torchfrom accelerate import Accelerator # +1accelerator = Accelerator() # +2model = torch.nn.Linear(10, 2)optimizer = torch.optim.Adam(model.parameters())dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3for epoch in range(10):for batch in dataloader:# No .to('cuda') needed - automatic!optimizer.zero_grad()loss = model(batch).mean()accelerator.backward(loss) # +4optimizer.step()
Configure (interactive):
accelerate config
Questions:
- Which machine? (single/multi GPU/TPU/CPU)
- How many machines? (1)
- Mixed precision? (no/fp16/bf16/fp8)
- DeepSpeed? (no/yes)
Launch (works on any setup):
# Single GPUaccelerate launch train.py# Multi-GPU (8 GPUs)accelerate launch --multi_gpu --num_processes 8 train.py# Multi-nodeaccelerate launch --multi_gpu --num_processes 16 \--num_machines 2 --machine_rank 0 \--main_process_ip $MASTER_ADDR \train.py
Workflow 2: Mixed precision training
Enable FP16/BF16:
from accelerate import Accelerator# FP16 (with gradient scaling)accelerator = Accelerator(mixed_precision='fp16')# BF16 (no scaling, more stable)accelerator = Accelerator(mixed_precision='bf16')# FP8 (H100+)accelerator = Accelerator(mixed_precision='fp8')model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)# Everything else is automatic!for batch in dataloader:with accelerator.autocast(): # Optional, done automaticallyloss = model(batch)accelerator.backward(loss)
Workflow 3: DeepSpeed ZeRO integration
Enable DeepSpeed ZeRO-2 (pass a DeepSpeedPlugin, not a raw dict):
from accelerate import Accelerator, DeepSpeedPlugindeepspeed_plugin = DeepSpeedPlugin(zero_stage=2, # ZeRO-2offload_optimizer_device="none", # or "cpu" to offloadgradient_accumulation_steps=4,)accelerator = Accelerator(mixed_precision='bf16',deepspeed_plugin=deepspeed_plugin, # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin]))# Same code as before!model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
Or point at a full DeepSpeed JSON config via the plugin:
from accelerate import Accelerator, DeepSpeedPlugin# hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict)deepspeed_plugin = DeepSpeedPlugin(hf_ds_config="ds_config.json")accelerator = Accelerator(mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin)
ds_config.json (a raw DeepSpeed config — passed via the plugin, NOT via --config_file):
{"fp16": {"enabled": false},"bf16": {"enabled": true},"zero_optimization": {"stage": 2,"offload_optimizer": {"device": "cpu"},"allgather_bucket_size": 5e8,"reduce_bucket_size": 5e8}}
Or via interactive config:
accelerate config# Select: DeepSpeed → ZeRO-2# This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml)
Launch (--config_file expects an accelerate YAML, not a raw DeepSpeed JSON):
# Uses the default accelerate config written by `accelerate config`accelerate launch train.py# Or point at a specific accelerate YAMLaccelerate launch --config_file accelerate_deepspeed.yaml train.py
Workflow 4: FSDP (Fully Sharded Data Parallel)
Enable FSDP:
from accelerate import Accelerator, FullyShardedDataParallelPluginfsdp_plugin = FullyShardedDataParallelPlugin(sharding_strategy="FULL_SHARD", # ZeRO-3 equivalentauto_wrap_policy="transformer_based_wrap", # valid: transformer_based_wrap | size_based_wrap | no_wrapcpu_offload=False)accelerator = Accelerator(mixed_precision='bf16',fsdp_plugin=fsdp_plugin)model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
Or via config:
accelerate config# Select: FSDP → Full Shard → No CPU Offload
Workflow 5: Gradient accumulation
Accumulate gradients:
from accelerate import Acceleratoraccelerator = Accelerator(gradient_accumulation_steps=4)model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)for batch in dataloader:with accelerator.accumulate(model): # Handles accumulationoptimizer.zero_grad()loss = model(batch)accelerator.backward(loss)optimizer.step()
Effective batch size: batch_size * num_gpus * gradient_accumulation_steps
When to use vs alternatives
Use Accelerate when:
- Want simplest distributed training
- Need single script for any hardware
- Use HuggingFace ecosystem
- Want flexibility (DDP/DeepSpeed/FSDP/Megatron)
- Need quick prototyping
Key advantages:
- 4 lines: Minimal code changes
- Unified API: Same code for DDP, DeepSpeed, FSDP, Megatron
- Automatic: Device placement, mixed precision, sharding
- Interactive config: No manual launcher setup
- Single launch: Works everywhere
Use alternatives instead:
- PyTorch Lightning: Need callbacks, high-level abstractions
- Ray Train: Multi-node orchestration, hyperparameter tuning
- DeepSpeed: Direct API control, advanced features
- Raw DDP: Maximum control, minimal abstraction
Common issues
Issue: Wrong device placement
Don't manually move to device:
# WRONGbatch = batch.to('cuda')# CORRECT# Accelerate handles it automatically after prepare()
Issue: Gradient accumulation not working
Use context manager:
# CORRECTwith accelerator.accumulate(model):optimizer.zero_grad()accelerator.backward(loss)optimizer.step()
Issue: Checkpointing in distributed
Use accelerator methods:
# Save only on main processif accelerator.is_main_process:accelerator.save_state('checkpoint/')# Load on all processesaccelerator.load_state('checkpoint/')
Issue: Different results with FSDP
Ensure same random seed:
from accelerate.utils import set_seedset_seed(42)
Advanced topics
Megatron integration: See references/megatron-integration.md for tensor parallelism, pipeline parallelism, and sequence parallelism setup.
Custom plugins: See references/custom-plugins.md for creating custom distributed plugins and advanced configuration.
Performance tuning: See references/performance.md for profiling, memory optimization, and best practices.
Hardware requirements
- CPU: Works (slow)
- Single GPU: Works
- Multi-GPU: DDP (default), DeepSpeed, or FSDP
- Multi-node: DDP, DeepSpeed, FSDP, Megatron
- TPU: Supported
- Apple MPS: Supported
Launcher requirements:
- DDP:
torch.distributed.run(built-in) - DeepSpeed:
deepspeed(pip install deepspeed) - FSDP: PyTorch 1.12+ (built-in)
- Megatron: Custom setup
Resources
- Docs: https://huggingface.co/docs/accelerate
- GitHub: https://github.com/huggingface/accelerate
- Version: 1.11.0+
- Tutorial: "Accelerate your scripts"
- Examples: https://github.com/huggingface/accelerate/tree/main/examples
- Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries