Repository navigation
Resident FLUX 3 and LTX streaming; image models in a 10.5 GiB GPU budget - #80
Merged
Merged
Conversation
resident_load("ltx") already forwards resident_gb to ltx23_load_pipeline;
stream_text passes stream to the Gemma 4 loader, so a serving entry can
use the recommend("ltx25") placement on 8-12 GB cards. pinned_bytes now
counts the streamed transformer blocks and encoder layers, which keep
their staging outside the handle's set.
Below 7 GB resident the text connectors stream too, and their sets live on the connector modules, outside the handle's staging. Disk-staged pairs, which hold no host copy, count as zero.
… VAE and tokenizer
…Image resident_gb and per-block gc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serving work for small and shared GPUs: the resident handle covers FLUX 3 and the small-card LTX-2.5 placement, and FLUX.2 klein, Z-Image, FLUX 3 and LTX-2.5 each fit a ~10.5 GiB GPU budget (a 16 GB card with a speech pair resident beside them). Contains all of feat/resident-ltx-stream (0.2.2.23) plus the image-budget work (0.2.2.24).
0.2.2.23: resident LTX streaming, FLUX 3, decode
resident_load("ltx", stream_text = TRUE)streams Gemma 4 layers;resident_gbpasses through, andpinned_bytescounts the streamed blocks, connector sets and encoder layers.resident_load(stage_from = "disk")stages the Gemma 4 encoder from disk too (refuses a non-Gemma 4 text encoder)..ltx23_video_array), so the host high-water is about 1.3x the result instead of several full-size float copies.resident_load("flux3")serves FLUX 3 Image Klein;flux3_load_pipeline(text_dir =, support_dir =)reads the text encoder, VAE and tokenizer from given directories so every file can come from an offline mount.0.2.2.24: image models in a 10.5 GiB budget
flux2_load_pipeline(stream_text = TRUE)andzimage_load_pipeline(stream_text = TRUE)stream the bf16 Qwen3 layers through the encode;resident_load()passesstream_textto flux2, flux3 and zimage.zimage_load_pipeline(resident_gb =)keeps that many GB of fp8 linears resident and streams the rest, as FLUX 3 does.diffuseR.block_gc); each block left 2-3 GB of temporaries until R collected them.Measured (16 GB card held to 10.5 GiB free, expandable segments, 1024x1024)
resident_gb = 4resident_gb = 4.38, diskA 3990-byte prompt: FLUX.2 klein unchanged, Z-Image
resident_gb = 48.8 GiB, FLUX 3 8 GB plan 6.3 GiB / 31 s. FLUX 3 on the 12 GB plan runs out of memory in this budget. Full numbers in NEWS.md.Tests
Run with
tinytest::run_test_file()on the touched areas, all passing: resident 47, sdxl 65, sd21 25, resident_ltx_stream 8, resident_flux3 14, dit_zimage 9, fp8_ltx23 20, txt2img_flux2 5, txt2img_zimage 5, zimage_transformer 6, zimage_revision 4, quantize_zimage 10, ltx23_video_array 9. The full suite was not run.