Repository navigation
LTX-2.5 distilled text-to-video - #71
Merged
Merged
Conversation
The 2.5 22B DiT is the 2.3 one minus the video feed-forward biases plus a learned keyframe marker. Both flags come from the checkpoint's embedded config via ltx23_dit_config_args(), which the bf16, fp8 and NF4 loaders now use (explicit ... overrides still win). - ltx23_feed_forward/ltx23_transformer_block/ltx23_transformer gain ff_bias, audio_ff_bias and use_keyframes_abs_pos_embedding (video_keyframes_mask in forward, applied after proj_in). - ltx23_open_checkpoint() accepts model_version 2.3* or 2.5*. - The NF4 TorchScript stack takes a zero-length bias as "no bias" and derives the weight rows from absmax and the input width. Verified against the real headers: 2.5 maps 4091/4091 DiT keys and 2.3 4186/4186, none unmapped or unfilled.
Review follow-ups to 08c6d80: - The unquantized transformer load moves out of ltx23_load_pipeline() into .ltx23_load_transformer(), which takes ... like the fp8/NF4 loaders, so a tiny config can exercise it. The test checks the 2.5 flags arrive through the embedded config and every parameter round trips exactly at float32. - Fixture files and quantization directories are created and removed inside local() scopes; the test asserts the source checkpoint is gone.
LTX-2.5 and later are distilled for the rectified-flow ancestral sampler (eta 1); 2.3 keeps plain Euler. txt2vid_ltx2() gains sampler = "auto" (chosen from the checkpoint's model_version, now kept on the pipeline), "euler" or "euler_ancestral", added last so positional calls are unaffected. - .ltx23_step(): the diffusers LTXEulerAncestralRFScheduler step; eta 0 is exactly the previous x + dt * v. - .ltx23_denoise() draws per-step noise (video, then audio) from seed + 10000 (stage 1) / seed + 20000 (stage 2) and restores the caller's RNG state, so the initial and stage-2 entry noise match the Euler run at the same seed. - Tests: Euler identity, last-step denoise, the marginal property (exact velocity at sigma steps to (1 - s_next) x0 + N(0, s_next^2)), version selection, and seeded/restoring loop behavior on a tiny DiT. - REFERENCES.md records sources for both 2.5 changes so far.
LTX-2.5 replaces Gemma 3 with a fine-tuned Gemma 4 12B (gemma4-12b-ltx-v1). The trunk matches Gemma 3's, but the norms scale by w (not 1 + w), values get a scale-free RMSNorm, attention scaling is 1, full layers use head_dim 512 with one KV head and k == v plus proportional RoPE, and each layer multiplies by a stored layer_scalar. - gemma4_text_model() ports the transformers Gemma4UnifiedTextModel text stack (fused SDPA, per-type causal/sliding masks with finite minima so left-pad rows stay finite). - load_gemma4_text_encoder() streams the text weights of the LTX-2.5 single file (config from its gemma_config metadata), skipping the vision/audio towers, the connectors' text projection and the embedded tokenizer assets; optional pinning as for Gemma 3. - gemma4_tokenizer()/encode_gemma4(): ranked-merge BPE with byte fallback (sharing the Qwen merge loop), read from tokenizer.json or the encoder file's tokenizer_json tensor. Gemma 4 adds no BOS. Left pads. - encode_with_gemma4(): the LTX-2.5 encode step; same output layout as encode_with_gemma3(). Tests: hidden states match transformers to 1.2e-5 on a tiny model with every 2.5 feature (mutating any one of norm form, layer_scalar, sliding window, proportional RoPE or the value path moves it by 0.4 to 30); tokenizer ids match HF tokenizers on synthetic cases and, at home, on the real 262k-vocab tokenizer (CJK, emoji, tabs, double spaces).
Review follow-ups to 38dd8ff: - R torch 0.17's torch_finfo() rejects bfloat16, so the default-dtype load failed on the first encode. The mask uses .gemma4_dtype_min(): bfloat16's finite minimum, torch_finfo() for the other float types. - encode_with_gemma4() strips with Python str.strip() semantics (.py_strip), removing Unicode spaces base trimws() keeps. - Tests: bf16 load and encode with fully padded rows stays finite and within 5% of float32; the bf16 mask minimum is finite in bf16; the stripping matches Python's whitespace set and " tell " encodes exactly like "tell".
The 12B encoder is 24 GB in bf16, which fits neither the 16 GB card nor the free host RAM. gemma4_quantize_nf4() streams the LTX-2.5 file and stores the 328 per-layer projections as NF4 (6.13 GB) with embeddings, norms and layer_scalar resident (2.01 GB). The artifact also keeps the connectors' text_embedding_projection (2.31 GB) and writes tokenizer.json, so the 26 GB source is not needed afterwards. load_gemma4_nf4() rebuilds the model (buffers included) and load_gemma4_text_encoder() dispatches to it for an artifact directory. Tests (tiny LTX-layout checkpoint, now carrying tokenizer_json): 40 casts; tokenizer from the file and from the artifact; projection kept, towers dropped; idempotent re-quantize; residents equal the source at the stored dtype; per-weight NF4 error under 0.15; the NF4 model equals a dense twin carrying its dequantized weights (< 1e-5); bf16 encode stays finite; a missing layer_scalar is a hard error; no temp leftovers.
- Flush before adding a tensor that would push the shard past shard_bytes (an oversized single tensor, like the 2 GB embedding, gets its own shard), and count residents at their stored element size, so the float32 fallback is sized correctly. - The manifest records tokenizer.json; reuse requires it along with the shards, so an artifact missing its tokenizer is rebuilt.
R's gc cannot see torch's native memory, so the dropped source tensors and NF4 temporaries accumulated between shard flushes: the real 12B run was OOM-killed at a 20 GB cgroup cap within 27 s. With a gc per tensor it completes in 110 s at 6.86 GB peak RSS (328 casts, 6 shards, 10.46 GB artifact).
LTX-2.5 ships its components as separate files, and the connectors' text projection lives in the Gemma 4 encoder file rather than with the transformer. ltx25_open_checkpoint() presents the transformer (single file or fp8/NF4 artifact), the projection (from the encoder file or its NF4 artifact), the convolutional video VAE (keys gain their vae. prefix) and the audio VAE + vocoder as one ltx23_checkpoint under the 2.3 key names, routing each get_tensor() to its source file. It refuses a non-2.5 transformer, the diffusion-decoder VAE, and an encoder file without the projection. - ltx23_load_pipeline() accepts an open ltx23_checkpoint. - txt2vid_ltx2() encodes with Gemma 4 when given a gemma4_tokenizer. - An encoder already loaded on the GPU encodes there instead of being sent to the CPU path. - The LTX fp8/NF4 quantizers collect garbage per tensor, as the Gemma 4 quantizer now does (R's gc cannot see torch's native memory).
ltx23_load_pipeline(gpu_components = "transformer") loads the NF4 transformer straight onto the compute device and leaves it out of the pinned host staging, so a long-lived LTX process no longer holds a second copy of it in RAM (LTX-2.5 at rest: 12.3 GB host RSS instead of about 23 GB). txt2vid_ltx2() treats such components as resident, runs the connectors where they are parked (their 2.3 GB text projection does not fit beside the 11 GB transformer on a 16 GB card), and the VAE decodes in 256 px tiles (512 px tiles OOM at 768x512x97; 256 px peak at 15.7 GB). Verified with a two-stage 768x512x97 LTX-2.5 render.
This reverts commit 704c95c.
gpu.ctl loads LTX as resident_load("ltx", checkpoint_path = <NF4
artifact>, text_encoder = <encoder artifact>, tokenizer = ...). For 2.5:
- ltx23_quantize_nf4() accepts an open checkpoint, so
ltx25_open_checkpoint() over the split files writes ONE self-contained
artifact (transformer, connectors and text projection, conv VAE, audio
VAE, vocoder), the shape of the 2.3 artifact. A transformer that is
already NF4 is bundled as is, so the artifact can be built without
the 42 GB bf16 source. Shards are named after the model version.
- resident_load() loads a Gemma 4 artifact with load_gemma4_text_encoder()
and gemma4_tokenizer() (the artifact carries its tokenizer), and the
Gemma 3 path otherwise.
Real bundle: 7196 tensors (1344 NF4), 10 shards, 19.52 GB, 7 min,
peak RSS 6.1 GB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds LTX-2.5 distilled text-to-video, run through the existing 2.3 pipeline.
What changes
ff_bias = FALSEand the keyframe position embedding from the embedded config. Real headers map 4091/4091 keys for 2.5 and 4186/4186 for 2.3.txt2vid_ltx2(sampler = "auto")uses the ancestral RF Euler sampler for 2.5 and Euler for 2.3.ltx25_open_checkpoint()presents 2.5's separate files as one 2.3-style checkpoint.ltx23_quantize_nf4()writes that checkpoint as one self-contained artifact, the same shape as the 2.3 artifact gpu.ctl already serves.resident_load("ltx")loads a Gemma 4 artifact.Not supported yet: the diffusion-decoder VAE, the duration head, and the temporal upsampler.
Verification
Fleet
The gpu.ctl entry would follow the 2.3 one:
resident_load("ltx", checkpoint_path = <ltx2.5-nf4 artifact>, text_encoder = <gemma4-nf4 artifact>, tokenizer = <the same gemma4-nf4 artifact>).The artifacts aren't uploaded to the hub, and the residency and RAM budget haven't been validated under gpu.ctl.