multi_gpu_quantization.md

Multi-GPU quantization

CUDA_VISIBLE_DEVICES controls which GPUs a process can see. It does not by itself enable parallel quantization. For a MoE model such as DeepSeek, select a balanced MoE device pool in QuantizeConfig. On a free-threaded Python build, set PYTHON_GIL=0 before starting Python to enable parallel expert quantization.

For example, to use physical GPUs 0 and 1:

CUDA_VISIBLE_DEVICES=0,1 PYTHON_GIL=0 python quantize_deepseek.py

Inside quantize_deepseek.py, configure the pool using the logical CUDA indices visible to that process (cuda:0 and cuda:1 in this example):

from gptqmodel import GPTQModel, QuantizeConfig
from gptqmodel.quantization.config import VramStrategy

model_id = "MODEL_REPO_OR_LOCAL_PATH"
quant_config = QuantizeConfig(
    bits=4,
    group_size=128,
    device="cuda:0",
    moe_vram_strategy=VramStrategy.BALANCED,
    moe_vram_strategy_devices=["cuda:0", "cuda:1"],
)

model = GPTQModel.load(model_id, quant_config, dtype="bfloat16")
model.quantize(calibration_dataset)
model.save("DeepSeek-GPTQ-4bit")

Provide calibration_dataset using the normal model.quantize() input format. The dense model modules remain on the default exclusive strategy; balanced MoE placement distributes expert work across the selected devices. The Python runtime must be a free-threaded build and the GIL must actually be disabled; PYTHON_GIL=0 cannot make a regular Python build free-threaded. GPT-QModel logs the visible dense/MoE device pools and whether parallel MoE quantization is eligible when quantization starts.

For a dense model, moe_vram_strategy does not apply. To distribute dense module work, configure dense_vram_strategy and dense_vram_strategy_devices instead. Loading an already-quantized model across GPUs is a separate operation and can use device_map="auto"; that controls inference placement, not quantization parallelism.