LTX-2.5 on 8 and 12 GB cards and in less host RAM

CLAUDE.md

@@ -495,6 +495,34 @@ params in fp32 before the bf16 cast needs 22 GB of host RAM, so the
 loader constructs a `.construct_skeleton()` (FLUX 3 opts in via
 `skeleton = TRUE` in `.flux_family_hooks`).
 
+### LTX on 8 and 12 GB cards
+
+`ltx23_load_pipeline(resident_gb = )` keeps the leading NF4 blocks on
+the GPU and streams the rest from pinned memory one block per
+TorchScript call (`.ltx23_block_stream`); under 7 GB the text
+connectors stream too. `load_gemma4_nf4(stream = TRUE)` streams Gemma 4
+layers. `ltx23_memory_profile()` picks `resident_gb = vram - 4` below
+14 GB. Simulated (ballast tensor) at 768x512x121: 12 GB and 8 GB both
+~84 s, same as 16 GB; PCIe 5 hides 6.7 GB/step of streaming.
+
+Gotcha that cost an hour: the first render in a process differs from
+later ones with the same seed (TorchScript's profiling executor swaps
+graphs after its first calls; max latent diff ~3), so compare streamed
+vs resident only after a warm-up run. Host RAM: the pipeline's pinned
+staging plus Gemma 4 need ~31 GB in one process. Low-RAM flow: encode
+with Gemma, free it, `ltx23_text_embeds()` on a connectors-only
+pipeline loaded `pin = FALSE`, free it, then load the render pipeline
+without connectors, optionally `resident_from = "disk"` (no host copy
+of the resident blocks; re-read at ~1 GB/s). Measured high-water marks
+at 768x512x121: text 12.4 GB; render 20.1 GB (host) / 16.0 GB (disk)
+on 8 GB, 14.9 GB (disk) on 12 GB. Set `MALLOC_MMAP_THRESHOLD_=1048576`
+so freed tensors return to the OS.
+
+Gotchas that bit once (host RAM): freed PINNED memory stays in torch's
+CachingHostAllocator, not the OS, so a pipeline you plan to drop must
+load unpinned (RSS stayed 9 GB after freeing pinned connectors). Pinned
+pages show up as RssShmem in /proc/self/status, not RssAnon.
+
 ### LTX-2.3 Video Generation (clean-room rewrite in progress)
 
 The original LTX-2.0 port was removed and is being replaced by a ground-up

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: diffuseR
 Title: Functional Interface to Diffusion Models in R
-Version: 0.2.2.17
+Version: 0.2.2.18
 Authors@R: c(
     person("Troy", "Hernandez", email = "troy@cornball.ai", role = c("aut", "cre"),
            comment = c(ORCID = "0009-0005-4248-604X")),

NAMESPACE

@@ -182,6 +182,7 @@ export(ltx23_split_keys)
 export(ltx23_stage2_distilled_sigmas)
 export(ltx23_tail_latents)
 export(ltx23_text_connectors)
+export(ltx23_text_embeds)
 export(ltx23_tone_map_latents)
 export(ltx23_transformer)
 export(ltx23_transformer_block)

NEWS.md

@@ -1,3 +1,49 @@
+# diffuseR 0.2.2.18
+
+* **LTX-2.5 on 8 and 12 GB cards.** The NF4 transformer can keep part
+  of itself on the GPU and stream the rest from pinned host memory, one
+  block per call, every step:
+  - `ltx23_load_pipeline(resident_gb = )` sets how much stays resident
+    (default `Inf`, all of it). Under 7 GB the text connectors (5.9 GB)
+    stream too: their two projections as units, their blocks one at a
+    time.
+  - `load_gemma4_nf4(stream = TRUE)` / `load_gemma4_text_encoder()`
+    stage Gemma 4 layer by layer, so an encode holds the embeddings and
+    one layer at a time (3 GB peak instead of 8.8 GB).
+  - `ltx23_memory_profile()` now returns `resident_gb` and
+    `stream_text`; its 10-14 GB and 7-10 GB tiers use NF4 with
+    streaming instead of fp8, which LTX-2.5 has no artifact for.
+  - Measured with the rest of a 16 GB card held back, 768x512x121:
+    ~84 s on 12 GB (8 GB resident, pipeline peak 9.6 GB) and on 8 GB
+    (4 GB resident, peak 5.7 GB), the same as the full card.
+* **LTX with less host RAM.** The full small-card flow in one process
+  needs about 31 GB (the pinned render pipeline plus Gemma 4). Two ways
+  to need less:
+  - `ltx23_text_embeds()` runs the connectors on a connectors-only
+    pipeline and returns the embeddings for `txt2vid_ltx2(connector_embeds = )`.
+    It takes raw `prompt_embeds`, so you encode with Gemma, free it, run
+    the connectors, free them, and only then load the render pipeline
+    (`components` without `"connectors"`). The text phase peaks at
+    12.4 GB.
+  - `ltx23_load_pipeline(resident_from = "disk")` keeps no host copy of
+    the GPU-resident NF4 blocks and re-reads them from the artifact each
+    render (~1 GB/s through safetensors).
+  - `ltx23_load_pipeline(pin = FALSE)` now also leaves the staging
+    buffers unpinned. Freed pinned memory stays in torch's host cache,
+    so a pipeline you will drop should load unpinned.
+  - `ltx23_memory_profile(host_ram_gb = )` (detected by default) returns
+    `split_text` (below 32 GB) and `resident_from` (`"disk"` below
+    22 GB).
+  - Measured at 768x512x121 under a 20 GB cgroup cap, split text phase,
+    second render. 8 GB card: host copies 20.1 GB high-water, 82 s;
+    disk 16.0 GB, 90 s. 12 GB card, disk: 14.9 GB (the load), 10.4 GB
+    while rendering, 92 s. Host and disk renders are bit-identical.
+* Streaming keeps the output: once TorchScript has warmed up, streamed
+  and resident renders are bit-identical. The first render in a process
+  still differs from later ones with the same seed (the profiling
+  executor switches graphs after its first calls); that predates this
+  release.
+
 # diffuseR 0.2.2.17
 
 * **FLUX 3 on 8 and 12 GB cards.** `flux3_load_pipeline()` plans from

R/connectors_ltx23.R

@@ -240,10 +240,19 @@ ltx23_connector_transformer_1d <- torch::nn_module(
 
     rotary_emb <- self$rope(batch_size, seq_len, device = hidden_states$device)
 
+    # Block streaming (see .ltx23_connector_stream)
+    stream <- self$layer_stream
     for (i in seq_along(self$transformer_blocks)) {
+        if (!is.null(stream)) {
+            .staged_onload(stream[[i]], hidden_states$device)
+        }
         hidden_states <- self$transformer_blocks[[i]](
             hidden_states, attention_mask = attention_mask, rotary_emb = rotary_emb
         )
+        if (!is.null(stream)) {
+            .staged_offload(stream[[i]])
+            gc(verbose = FALSE, full = FALSE)
+        }
     }
     hidden_states <- self$norm_out(hidden_states)
 
@@ -355,8 +364,11 @@ ltx23_text_connectors <- torch::nn_module(
     video_norm <- norm_states$mul(sqrt(self$video_hidden_dim / self$caption_channels))
     audio_norm <- norm_states$mul(sqrt(self$audio_hidden_dim / self$caption_channels))
 
-    video_proj <- self$video_text_proj_in(video_norm)
-    audio_proj <- self$audio_text_proj_in(audio_norm)
+    video_proj <- .unit_call(self, "video_text_proj_in", video_norm$device,
+                             function() self$video_text_proj_in(video_norm))
+    audio_proj <- .unit_call(self, "audio_text_proj_in", audio_norm$device,
+                             function() self$audio_text_proj_in(audio_norm))
+    rm(norm_states, video_norm, audio_norm)
 
     # Multiplicative [B, S] mask -> additive [B, 1, 1, S] with -finfo max
     text_dtype <- video_proj$dtype

R/dit_ltx23.R

@@ -368,23 +368,58 @@ ltx23_transformer <- torch::nn_module(
     !is.null(video_ca_rotary_emb) && !is.null(audio_ca_rotary_emb) &&
     .ltx23_jit_block_ok(self$transformer_blocks[[1]])
 
+    # Block streaming (see .ltx23_block_stream): blocks past n_resident
+    # live in pinned host memory and visit the device one at a time
+    stream <- self$block_stream
+    n_blocks <- length(self$transformer_blocks)
+    n_resident <- if (is.null(stream)) n_blocks else stream$n_resident
+    stream_on <- function(i) {
+        if (i > n_resident) {
+            .staged_onload(stream$staging[[i - n_resident]],
+                           hidden_states$device)
+        }
+    }
+    stream_off <- function(i) {
+        if (i > n_resident) {
+            .staged_offload(stream$staging[[i - n_resident]])
+            # The onload's R-side copies hold the device storage until
+            # collected
+            gc(verbose = FALSE, full = FALSE)
+        }
+    }
+
     if (use_jit) {
-        res <- .ltx23_jit_run_stack(
-                                    self$transformer_blocks,
-                                    hidden_states, audio_hidden_states,
-                                    encoder_hidden_states, audio_encoder_hidden_states,
-                                    temb, temb_audio,
-                                    video_ca_scale_shift, audio_ca_scale_shift,
-                                    video_ca_a2v_gate, audio_ca_v2a_gate,
-                                    temb_prompt, temb_prompt_audio,
-                                    video_rotary_emb, audio_rotary_emb,
-                                    video_ca_rotary_emb, audio_ca_rotary_emb,
-                                    encoder_attention_mask,
-                                    audio_encoder_attention_mask,
-                                    cond_token_index = cond_token_index
-        )
-        hidden_states <- res[[1]]
-        audio_hidden_states <- res[[2]]
+        run_stack <- function(blocks) {
+            res <- .ltx23_jit_run_stack(
+                                        blocks,
+                                        hidden_states, audio_hidden_states,
+                                        encoder_hidden_states, audio_encoder_hidden_states,
+                                        temb, temb_audio,
+                                        video_ca_scale_shift, audio_ca_scale_shift,
+                                        video_ca_a2v_gate, audio_ca_v2a_gate,
+                                        temb_prompt, temb_prompt_audio,
+                                        video_rotary_emb, audio_rotary_emb,
+                                        video_ca_rotary_emb, audio_ca_rotary_emb,
+                                        encoder_attention_mask,
+                                        audio_encoder_attention_mask,
+                                        cond_token_index = cond_token_index
+            )
+            hidden_states <<- res[[1]]
+            audio_hidden_states <<- res[[2]]
+        }
+        if (n_resident == n_blocks) {
+            run_stack(self$transformer_blocks)
+        } else {
+            if (n_resident > 0L) {
+                run_stack(lapply(seq_len(n_resident),
+                                 function(i) self$transformer_blocks[[i]]))
+            }
+            for (i in seq(n_resident + 1L, n_blocks)) {
+                stream_on(i)
+                run_stack(list(self$transformer_blocks[[i]]))
+                stream_off(i)
+            }
+        }
     }
 
     block_gc <- isTRUE(getOption("diffuseR.block_gc"))
@@ -402,6 +437,7 @@ ltx23_transformer <- torch::nn_module(
     eager_blocks <- if (use_jit) integer(0) else seq_along(self$transformer_blocks)
     for (block_idx in eager_blocks) {
         is_stg_block <- (block_idx - 1L) %in% stg_blocks # reference indices are 0-based
+        stream_on(block_idx)
         res <- self$transformer_blocks[[block_idx]](
                                       hidden_states = hidden_states,
                                       audio_hidden_states = audio_hidden_states,
@@ -429,6 +465,7 @@ ltx23_transformer <- torch::nn_module(
         )
         hidden_states <- res[[1]]
         audio_hidden_states <- res[[2]]
+        stream_off(block_idx)
         if (isTRUE(getOption("diffuseR.debug"))) {
             ms <- torch::cuda_memory_stats()
             message(sprintf("    block %d: %.2f GB allocated, %.2f GB reserved",

R/gemma4_text_encoder.R

@@ -217,13 +217,23 @@ gemma4_text_model <- torch::nn_module(
                    dtype = dtype)
 
     all_hidden <- NULL
+    # Layer streaming (load with stream = TRUE): each layer's weights
+    # visit the device only for its own forward
+    stream <- self$layer_stream
     for (i in seq_along(self$layers)) {
         if (output_hidden_states) {
             all_hidden <- c(all_hidden, list(hidden_states))
         }
         type <- self$config$layer_types[[i]]
+        if (!is.null(stream)) {
+            .staged_onload(stream[[i]], device)
+        }
         hidden_states <- self$layers[[i]](hidden_states, masks[[type]],
                                           rope[[type]][[1]], rope[[type]][[2]])
+        if (!is.null(stream)) {
+            .staged_offload(stream[[i]])
+            gc(verbose = FALSE, full = FALSE)
+        }
     }
     hidden_states <- self$norm(hidden_states)
     if (output_hidden_states) {
@@ -342,17 +352,19 @@ gemma4_text_model <- torch::nn_module(
 #'   \code{\link{encode_with_gemma4}} can move the model to the GPU per
 #'   encode at DMA speed. Default follows
 #'   \code{options(diffuseR.pin_staging)}.
+#' @param stream Logical. Stage layer by layer (see
+#'   \code{\link{load_gemma4_nf4}}).
 #' @param verbose Logical.
 #'
 #' @return A \code{gemma4_text_model}.
 #'
 #' @export
 load_gemma4_text_encoder <- function(path, device = "cpu", dtype = "bfloat16",
                                      pin = getOption("diffuseR.pin_staging", TRUE),
-                                     verbose = TRUE) {
+                                     stream = FALSE, verbose = TRUE) {
     if (file.exists(file.path(path, "manifest.json"))) {
         return(load_gemma4_nf4(path, device = device, dtype = dtype,
-                               pin = pin, verbose = verbose))
+                               pin = pin, stream = stream, verbose = verbose))
     }
     ckpt <- ltx23_open_checkpoint(path, require_version = NULL)
     meta <- ckpt$handle$metadata[["__metadata__"]]
@@ -383,7 +395,10 @@ load_gemma4_text_encoder <- function(path, device = "cpu", dtype = "bfloat16",
     model$to(device = device)
     model$eval()
 
-    if (pin && device == "cpu") {
+    if (stream && device == "cpu") {
+        attr(model, "staging") <- .stage_layer_stream(model, model$layers,
+                                                      "layers.", pin)
+    } else if (pin && device == "cpu") {
         st <- .pin_component(model)
         if (!is.null(st)) {
             attr(model, "staging") <- st