Skip to content

FLUX 3 in less host RAM: fp8 Qwen3-VL text encoder - #79

Merged
TroyHernandez merged 2 commits into
mainfrom
feat/flux3-qwen3vl-quant
Oct 29, 2026
Merged

TroyHernandez merged 2 commits into
mainfrom
feat/flux3-qwen3vl-quant

Conversation

@TroyHernandez

Copy link
Copy Markdown
Contributor

FLUX 3 now ships an fp8 Qwen3-VL text encoder artifact, which lowers peak host RAM from 20.2 GB to 15.5 GB at 1024x1024 with the same speed and near-identical images.

Changes

  • R/quantize_qwen3.R (new): qwen3_quantize() writes a Qwen3 or Qwen3-VL text model as an fp8 or NF4 artifact. The 252 projection linears are quantized; embeddings and norms stay bf16; the vision tower and tied LM head are dropped.
  • load_qwen3_text_encoder() loads such an artifact. Its config-to-args code moved into .qwen3_args().
  • download_flux3_klein(text_precision = "fp8") builds the artifact by default. flux3_load_pipeline(text_precision = "auto") uses fp8, then NF4, else bf16.
  • Staging collects the fp8 weight fields for the text encoder, both whole-component and per streamed layer.
  • The fp8 tensor cast is shared as .fp8_quantize().

Measured (16 GB card, 1024x1024, two renders each)

Text encoder Host RAM peak Time Image vs bf16
bf16 20.2 GB ~24 s reference
fp8 15.5 GB ~24 s 27.2 dB PSNR
NF4 13.5 GB ~25 s 15.1 dB (different composition)
  • Prompt embeddings differ from bf16 by 2.7% with fp8 and about 5% with NF4, so NF4 is opt-in.
  • With the 8 GB streaming plan and fp8 text, the GPU peak is 4.9 GB and the image is pixel-identical to the resident run.

Full tinytest suite: 1745 passed, 0 failed (37 new).

@TroyHernandez
TroyHernandez merged commit 52d5d78 into main Oct 29, 2026
2 checks passed
@TroyHernandez
TroyHernandez deleted the feat/flux3-qwen3vl-quant branch October 29, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant