ascend-ai-coding/awesome-ascend-skills/mindspeed-mm-vlm

MindSpeed-MM VLM (Vision-Language Model) Training

This Skill guides users through training multimodal understanding (VLM) models on Huawei Ascend NPU using MindSpeed-MM. It uses Qwen2.5VL-3B as the flagship example and covers the end-to-end fine-tuning workflow.

Prerequisites

Critical: For most VLMs (Qwen2.5VL, Qwen2VL, InternVL, GLM4V, DeepSeekVL2), follow the manual install flow below. Do NOT use bash scripts/install.sh — official MindSpeed-MM docs state it only fully supports Qwen3/Qwen3.5. (For Qwen3VL / Qwen3.5, use one-click install + bash examples/qwen3_5/install_extensions.sh.)

Step P1: Clone repositories

git clone https://gitcode.com/Ascend/MindSpeed-MM.git /root/workspace/MindSpeed-MM
git clone https://github.com/NVIDIA/Megatron-LM.git /root/workspace/Megatron-LM
cd /root/workspace/Megatron-LM && git checkout core_v0.12.1
cp -r megatron /root/workspace/MindSpeed-MM/
cd /root/workspace/MindSpeed-MM

Step P2: Install PyTorch + torch_npu

Always use pre-built wheels from Ascend PyTorch Releases. Do not use pip install torch==2.7.1 — aarch64 wheels on PyPI are unreliable.

# Download matching wheels from https://gitcode.com/Ascend/pytorch/releases
# Replace cp310 with cp311 if using Python 3.11
pip install torch-2.7.1-cp310-cp310-manylinux_2_28_aarch64.whl
pip install torch_npu-2.7.1rc1-cp310-cp310-manylinux_2_28_aarch64.whl
# For x86_64: same naming pattern with _x86_64 suffix
 
# Required by torch_npu and other components:
pip install numpy pyyaml scipy attrs decorator psutil

Step P3: Install MindSpeed

git clone https://gitcode.com/Ascend/MindSpeed.git /root/workspace/MindSpeed
cd /root/workspace/MindSpeed
git checkout 93c45456c7044bacddebc5072316c01006c938f9
pip install -r requirements.txt
pip install -e .
cd /root/workspace/MindSpeed-MM

Step P4: Install MindSpeed-MM base dependencies

# Installs base stack from pyproject.toml: transformers==4.57.0, diffusers==0.30.3, peft==0.7.1, etc.
pip install -e .

Step P5: Activate CANN environment

source /usr/local/Ascend/cann/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

Step P6: Smoke test

python3 -c "
import torch, torch_npu
assert torch.npu.is_available(), 'NPU not available'
print(f'NPU count: {torch_npu.npu.device_count()}')
import transformers, diffusers, mindspeed
print(f'transformers={transformers.__version__}, diffusers={diffusers.__version__}')
"

Then proceed to Step 0 below for model-specific dependencies. For Qwen2.5VL the base is sufficient; other models need overlays (see Step 0 table).

For detailed troubleshooting and alternative install paths, see mindspeed-mm-env-setup.

Supported VLM Models

Entry Script Note: VLM models use different entry scripts. Check the shell script in examples/<model_name>/ for the exact command — do not assume from the model name.

How to Train Any VLM Model

The workflow for training any VLM model in MindSpeed-MM follows a universal pattern:

1. Find the example: Look in examples/<model_name>/ for available shell scripts (finetune, pretrain, inference, evaluate)
2. Read the shell script: Identify the entry script, config files, and parallelism settings
3. Check weight conversion: Determine if the model needs mm-convert (Megatron models), DCP conversion (FSDP2 models), or uses HF weights directly (custom trainer models)
4. Modify config files: Update data paths and weight paths in the config files (JSON or YAML depending on the framework)
5. Launch the shell script: bash examples/<model_name>/finetune_<model>_<size>.sh

For other models, adapt the Qwen2.5VL quick start below. Use the Model Registry to look up the exact entry script, converter, and backend for your target model.

Framework Patterns

All VLM models in MindSpeed-MM follow one of three framework patterns:

Pattern 1: Megatron (Qwen2.5VL, Qwen2VL, InternVL2.5, InternVL3, GLM4.1V, DeepSeekVL2)

# Uses JSON config files: data.json + model.json
# Weight conversion: mm-convert <Converter> hf_to_mm
torchrun $DISTRIBUTED_ARGS pretrain_vlm.py \
  --mm-data examples/<model>/data.json \
  --mm-model examples/<model>/model.json \
  --load $MM_WEIGHT_PATH ...

Pattern 2: FSDP2 (Qwen3VL, InternVL3.5, GLM4.5V)

# Uses YAML config file; requires CUDA_DEVICE_MAX_CONNECTIONS=2
# Weight conversion: mm-convert <Converter> hf_to_dcp
# MoE models may need: --init-model-with-meta-device
CUDA_DEVICE_MAX_CONNECTIONS=2 torchrun $DISTRIBUTED_ARGS pretrain_transformers.py \
  --fsdp2-config-path examples/<model>/fsdp2_config.yaml ...

Pattern 3: Custom Trainers (DeepSeekOCR, DeepSeekOCR2, Ming)

# Standalone entry scripts; uses HF weights directly (no conversion needed)
torchrun $DISTRIBUTED_ARGS examples/<model>/finetune_<model>.py <args>

Quick Start: Qwen2.5VL-3B Fine-Tuning (End-to-End)

This is the flagship Megatron-pattern example. For other models, follow the same workflow structure but use configs and scripts from examples/<model_name>/. FSDP2 models use YAML configs instead of JSON; Custom trainer models skip weight conversion entirely. See the "Framework Patterns" section above.

Step 0: Model-Specific Dependencies

Different VLM models have different version requirements for libraries such as transformers. Some of these requirements conflict, so environment isolation is necessary.

Important: Qwen3VL and DeepSeekOCR have conflicting transformers version requirements. If you need to support both, use separate Docker containers or conda environments. See per-model-deps.md for details.

Step 1: Weight Download and Conversion

1.1 Download HuggingFace Weights

# Download from HuggingFace (requires proxy)
http_proxy=http://127.0.0.1:58232 https_proxy=http://127.0.0.1:58232 \
  huggingface-cli download Qwen/Qwen2.5-VL-3B-Instruct \
  --local-dir ckpt/hf_path/Qwen2.5-VL-3B-Instruct

1.2 HF to MM Format Conversion

MindSpeed-MM uses its own weight format (MM format). Weights must be converted before training.

mm-convert Qwen2_5_VLConverter hf_to_mm \
  --cfg.mm_dir "ckpt/mm_path/Qwen2.5-VL-3B-Instruct" \
  --cfg.hf_config.hf_dir "ckpt/hf_path/Qwen2.5-VL-3B-Instruct" \
  --cfg.parallel_config.llm_pp_layers [[36]] \
  --cfg.parallel_config.vit_pp_layers [[32]] \
  --cfg.parallel_config.tp_size 1

Key parameter descriptions:

Important: llm_pp_layers and vit_pp_layers must match the pipeline_num_layers in model.json. When PP > 1, you need to split the layer counts, e.g., [[18,18]] for PP=2.

1.3 Converters for Different Models

Step 2: Dataset Preparation

VLM training uses the MLLM JSON format. This example uses COCO2017 + LLaVA-Instruct-150K.

2.1 Download Data

# 1. Download COCO2017 training images
mkdir -p data/COCO2017/train2017
# Extract to data/COCO2017/train2017/
 
# 2. Download LLaVA-Instruct-150K
# Download llava_instruct_150k.json to data/

2.2 Convert to MLLM Format

Important: The conversion script uses hardcoded relative paths. It expects ./data/llava_instruct_150k.json as input and ./data/COCO2017/train2017/ for image lookup. Run it from the MindSpeed-MM root directory, and ensure data is at ./data/ (or create symlinks).

cd /root/workspace/MindSpeed-MM  # Must run from repo root
python examples/qwen2vl/llava_instruct_2_mllm_demo_format.py

Output: data/mllm_format_llava_instruct_data.json

The script converts LLaVA format (conversations/image) to MLLM format (messages/images), which is what MindSpeed-MM expects.

2.3 Data Directory Structure

data/
├── COCO2017/train2017/          # Image files
│   ├── 000000000009.jpg
│   ├── 000000000025.jpg
│   └── ...
├── llava_instruct_150k.json     # Original data (for conversion)
└── mllm_format_llava_instruct_data.json  # MLLM format (for training)

Field name matching: The field names in your data JSON must match the attr section in data.json config. Default mapping: messages for conversations, images for image paths. Using different field names (e.g., conversations instead of messages) will cause KeyError.

For detailed MLLM JSON format specifications, see data-format.md.

Step 3: Configuration Files

Qwen2.5VL-3B fine-tuning requires two JSON configuration files:

3.1 data.json (Data Configuration)

Path: examples/qwen2.5vl/data_3b.json