yuhuabillchen/justfit-mlx/convert-quantize

Convert & Quantize

Use this workflow for mlx_vlm.convert — turning a Hugging Face checkpoint into MLX format, optionally quantizing it.

First Checks

1. Confirm the source: a Hugging Face repo id or a local path (--hf-path, alias --model).
2. Confirm the model family is supported in mlx_vlm/models/ (a folder named after the config.json model_type). If not, this is a porting task — switch to Skill("mlx-vlm-skills:add-new-model").
3. Verify current flags before finalizing: uv run mlx_vlm.convert --help.
4. Entry point is mlx_vlm.convert. python -m mlx_vlm.convert is deprecated; use uv run mlx_vlm.convert ... or python -m mlx_vlm convert ....

Command Patterns

Plain convert (no quantization), saves to ./mlx_model by default:

uv run mlx_vlm.convert --hf-path <repo-or-path> --mlx-path ./out-mlx

4-bit affine quantization (RTN, the default method):

uv run mlx_vlm.convert --hf-path <repo-or-path> --mlx-path ./out-4bit -q --q-bits 4 --q-group-size 64

Other quant modes (--q-mode sets its own bit/group defaults):

# mxfp4 (group 32, 4 bit), nvfp4 (group 16, 4 bit), mxfp8 (group 32, 8 bit)
uv run mlx_vlm.convert --hf-path <repo-or-path> --mlx-path ./out-mxfp4 -q --q-mode mxfp4

Mixed-bit recipe (per-layer bit allocation, llama.cpp-style):

# recipes: mixed_2_6 mixed_3_4 mixed_3_5 mixed_3_6 mixed_3_8 mixed_4_6 mixed_4_8
uv run mlx_vlm.convert --hf-path <repo-or-path> --mlx-path ./out-mixed -q --quant-predicate mixed_3_6

AWQ (activation-aware, needs a calibration pass):

uv run mlx_vlm.convert --hf-path <repo-or-path> --mlx-path ./out-awq -q --quant-method awq \
  --calibration multimodal --calibration-data /path/to/media   # or --calibration text (default)

dtype cast only / dequantize:

uv run mlx_vlm.convert --hf-path <repo-or-path> --mlx-path ./out-bf16 --dtype bfloat16
uv run mlx_vlm.convert --hf-path <quantized-repo> --mlx-path ./out-fp -d   # dequantize

Upload the result to the Hub:

uv run mlx_vlm.convert --hf-path <repo> --mlx-path ./out -q --upload-repo <user>/<name>-mlx

Key Facts

• Multimodal modules are skipped from quantization by default (skip_multimodal_module) — vision/audio towers stay full precision; only the language model is quantized. This is expected, not a bug.
• --q-mode choices: affine (default, group 64 / 4 bit), mxfp4 (32/4), nvfp4 (16/4), mxfp8 (32/8). --q-bits/--q-group-size override the mode defaults.
• --quant-method: rtn (default, round-to-nearest) or awq (needs calibration; --calibration text|multimodal, optional --calibration-data).
• -q/--quantize and -d/--dequantize are mutually exclusive.
• --dtype (from config.json torch_dtype if unset) casts float weights; useful for shrinking fp32 → bf16 without quantizing.
• The converted folder gets the weights, copied *.py/*.json, the processor, a regenerated config.json, and a model card — it is ready for mlx_vlm.generate and the server.

Validation

• After converting, load and run a tiny generation to prove the checkpoint works — see Skill("mlx-vlm-skills:cli-inference"). Run on Apple Silicon with enough RAM for the model; do not attempt a large model on an 8 GB machine.
• For quality, compare a few greedy outputs between the source and the quantized model; large drops usually mean too-low bits or a wrong --q-mode for the model.
• For code changes touching conversion, run uv run --with pytest python -m pytest mlx_vlm/tests/test_utils.py -q and any model-specific test.
• If the outcome is a bug report, switch to Skill("mlx-vlm-skills:reproducible-github-issues").