Add LTX-2.5 DFR pipeline (keyframe slots, spatial detailing, tiled temporal rounds)

docs/source/en/api/pipelines/ltx2.md

@@ -1030,6 +1030,117 @@ encode_video(
 
 You can see the supported workflows in the docs for each blockset (e.g. [`LTX2AutoBlocks`], [`LTX25AutoBlocks`]).
 
+### Diffusion Fidelity Rendering (DFR) for LTX-2.5
+
+`LTX2DFRPipeline` trades wall-clock time for detail fidelity. It generates at a fraction of the requested resolution *plus* extra single-pixel-frame **keyframe slots**, then re-renders at the full resolution seeded from both. A slot costs a full latent frame of tokens to buy one pixel frame, which relaxes the effective temporal compression at that position — so the surrounding video is conditioned on genuinely new frames instead of interpolated ones. Slot positions come from a segment grid aligned to the VAE's temporal border (24 or 32 pixel frames, whichever pads the request less); the canvas is padded to a whole number of segments internally and trimmed back before decoding.
+
+This needs a transformer whose config sets `use_keyframes_abs_pos_embedding`, which marks single-pixel-frame latents with a learned embedding. LTX-2.5 checkpoints ship it; the pipeline raises on anything older rather than spending the token budget on tokens the model cannot interpret.
+
+Budget for the extra tokens: each slot adds one latent frame's worth, so stage 2 runs a longer sequence than the equivalent two-stage distilled pass — +31% at 1024x1536 / 121 frames (24576 -> 32256 tokens, 5 slots on a 24-frame segment grid). Peak activation memory scales with that, so a resolution that just fits the plain distilled recipe may need `enable_sequential_cpu_offload`, `vae.enable_tiling()`, or a smaller canvas under DFR.
+
+The full recipe below is the one worth starting from: 1088x1920 image-to-video, one x2 temporal refine round,
+and the x2 spatial detailing IC-LoRA on stage 2. The individual knobs are explained after it.
+
+```py
+import torch
+from diffusers import LTX2DFRPipeline
+from diffusers.pipelines.ltx2 import LTX2LatentUpsamplerModel
+from diffusers.pipelines.ltx2.pipeline_ltx2_condition import LTX2VideoCondition
+from diffusers.utils import encode_video, load_image
+
+# The published `model_index.json` has no upsamplers. The spatial one is a subfolder of the repo; the x2 temporal
+# one is not published — convert it with `--temporal_latent_upsampler` (see "Temporal refinement" below).
+latent_upsampler = LTX2LatentUpsamplerModel.from_pretrained(
+    "Lightricks/LTX-2.5-Diffusers", subfolder="latent_upsampler", torch_dtype=torch.bfloat16
+)
+temporal_latent_upsampler = LTX2LatentUpsamplerModel.from_pretrained(
+    "path/to/converted/temporal_latent_upsampler", torch_dtype=torch.bfloat16
+)
+pipe = LTX2DFRPipeline.from_pretrained(
+    "Lightricks/LTX-2.5-Diffusers",
+    latent_upsampler=latent_upsampler,
+    temporal_latent_upsampler=temporal_latent_upsampler,
+    torch_dtype=torch.bfloat16,
+)
+
+# Stage 2 attends to the stage-1 latent through this IC-LoRA. It is calibrated for strength 0.5.
+pipe.load_lora_weights("Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler", adapter_name="detailing")
+pipe.set_adapters(["detailing"], weights=[0.5])
+pipe.enable_model_cpu_offload()
+
+image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
+frame_rate = 24.0
+video, audio = pipe(
+    prompt="A tabby cat stretching in a sunlit window, dust motes drifting in the light",
+    conditions=[LTX2VideoCondition(frames=image, index=0, strength=1.0)],
+    height=1088,
+    width=1920,
+    frame_rate=frame_rate,
+    # `num_frames` omitted: the `duration_head` predicts it from the prompt.
+    temporal_upscalings=1,
+    detailing_lora_adapter_name="detailing",
+    generator=torch.Generator(device="cuda").manual_seed(0),
+    output_type="np",
+    return_dict=False,
+)
+
+# One refine round doubles the playback rate.
+playback_fps = frame_rate * 2**1
+encode_video(
+    video[0],
+    fps=playback_fps,
+    audio=audio[0].float().cpu(),
+    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
+    output_path="ltx2_5_dfr.mp4",
+)
+```
+
+`height` and `width` are the *output* resolution and must be divisible by `2 ** spatial_upscalings` times the VAE's spatial compression ratio (64 for LTX-2.5 at the default `spatial_upscalings=1`, 128 at `spatial_upscalings=2`), since every stage below the output runs at a halved resolution. This is why a 4K run is **3840x2176**, not 3840x2160: UHD height has to be snapped up onto the 128 grid. 1080p is likewise 1920x1088, not 1920x1080. Each stage runs its own fixed distilled schedule (`stage_1_sigmas`, `stage_2_sigmas`), so there is no `num_inference_steps`; the distilled schedules are trained to be used without guidance, so there is no `negative_prompt` or `guidance_scale` either. The shipped audio is stage 1's — stage 2 and the temporal refine tiles still run an audio pass so the video branch has cross-modal attention; the waveform itself is not refined after stage 1.
+
+**Spatial detailing.** Load the 2x spatial detailing IC-LoRA under a named adapter and pass that name as `detailing_lora_adapter_name`. Stage 2 then runs with the adapter active *and* attends to the stage-1 half-resolution latent as an in-context reference; stage 1 and the temporal rounds run with it deactivated, and whatever adapters were active on entry are restored before the call returns.
+
+`detailing_reference_downscale_factor` (default `2`) scales the reference tokens' spatial coordinates into the target's coordinate space, and should match the LoRA's `reference_downscale_factor` metadata. The pipeline only activates and deactivates the adapter — its weight stays whatever you set, so give it the 0.5 it is calibrated for.
+
+**Temporal refinement.** `temporal_upscalings` (0-2) adds rounds that each double the frame rate: the canvas is temporally upsampled, split into `2 ** round` tiles that meet at shared keyframes, given fresh mid-segment slots, and densified with ancestral Euler. Each tile is handed the slice of the frozen stage-1 audio covering its own playback window, resampled to the tile's token count, so both sides of a seam densify against the same sound. Rounds need the optional `temporal_latent_upsampler` component, which is not part of the base repo — convert it with `--temporal_latent_upsampler` and pass it to `from_pretrained`, as in the example above.
+
+You always get `(num_frames - 1) * 2 ** temporal_upscalings + 1` frames back, and the returned video plays at `frame_rate * 2 ** temporal_upscalings`. Conditioning fps is 60 whenever playback is above 30, independently of muxing: RoPE time is `pixel_frame / fps`, so a 120 fps time base would halve every token's temporal span versus the trained distribution, and 48 fps would stretch it. Both lie that they are 60 and treat the decoded frames at the playback rate.
+
+**A third spatial stage.** `spatial_upscalings=2` starts the base canvas one more factor of two down and adds a detailing pass at the output resolution *after* the temporal rounds. A full-resolution forward pass over the refined canvas does not fit in one sequence, so the pass denoises the whole canvas in a single loop and tiles the *transformer call* inside it — two ways on each spatial axis and `2 ** temporal_upscalings` ways in time. Every Euler step therefore steps a canvas whose tiles have already agreed on their overlaps, and conditionings are attached once on the whole canvas and filtered per tile at the token level rather than cropped by hand.
+
+The two axes are seamed differently. Neither side of a spatial border holds a known answer, so those overlaps are blended with trapezoidal weights. The temporal tiles are cut instead, on the same keyframe seams the last refine round stitched on: both windows reproduce a shared keyframe there, so the later one drops its run-up rather than averaging it. The keyframes the rounds settled are handed to the epilogue as content — each is decoded on its own, stretched x2 with Lanczos in RGB, encoded again at the output resolution, and pinned fully clean — so nothing along a seam is generated twice. Since the epilogue is a detailing pass, pass `detailing_lora_adapter_name` with it:
+
+```py
+video, audio = pipe(
+    prompt=prompt,
+    height=1024,
+    width=1920,  # both divisible by 128 at spatial_upscalings=2
+    num_frames=121,
+    spatial_upscalings=2,
+    detailing_lora_adapter_name="detailing",
+    output_type="np",
+    return_dict=False,
+)
+```
+
+**Decoding with the diffusion decoder.** `LTX2DFRPipeline` decodes with its convolutional `vae`. For maximum detail fidelity, run with `output_type="latent"` and hand the latents to [`LTX2VideoDiffusionDecodePipeline`].
+
+```py
+from diffusers import LTX2VideoDiffusionDecodePipeline
+from diffusers.models.autoencoders.ltx2_diffusion_decoder import LTX2VideoDiffusionDecoderModel
+
+video_latents, audio_latents = pipe(prompt=prompt, output_type="latent", return_dict=False)
+
+decoder = LTX2VideoDiffusionDecoderModel.from_pretrained(
+    "Lightricks/LTX-2.5-Diffusers", subfolder="diffusion_decoder", dtype=torch.bfloat16
+)
+decode_pipe = LTX2VideoDiffusionDecodePipeline(
+    diffusion_decoder=decoder, scheduler=pipe.scheduler, vae=pipe.vae
+)
+decode_pipe.enable_model_cpu_offload()
+# `denormalize=False`: the `output_type="latent"` path already applied the latent statistics.
+video = decode_pipe(latents=video_latents, denormalize=False, output_type="np", return_dict=False)[0]
+```
+
 ## LTX2Pipeline
 
 [[autodoc]] LTX2Pipeline

scripts/convert_ltx2_to_diffusers.py

@@ -443,7 +443,7 @@ def get_ltx2_transformer_config(version: str) -> tuple[dict[str, Any], dict[str,
                 "use_prompt_embeddings": False,
                 "perturbed_attn": True,
                 # The only transformer-level deltas from 2.3: the video FFN drops its bias (audio_ff_bias and
-                # use_prompt_adaln_single keep their True defaults for this checkpoint), and 2.5.1+ carries a
+                # use_prompt_adaln_single keep their True defaults for this checkpoint), and 2.5 carries a
                 # learned keyframe absolute-position embedding.
                 "ff_bias": False,
                 "use_keyframes_abs_pos_embedding": True,
@@ -1222,7 +1222,7 @@ def get_ltx2_spatial_latent_upsampler_config(version: str):
             "rational_spatial_scale": 2.0,
             "use_rational_resampler": True,
         }
-    elif version == "2.3":
+    elif version in ("2.3", "2.5"):
         config = {
             "in_channels": 128,
             "mid_channels": 1024,
@@ -1238,9 +1238,21 @@ def get_ltx2_spatial_latent_upsampler_config(version: str):
     return config
 
 
-def convert_ltx2_spatial_latent_upsampler(
-    original_state_dict: dict[str, Any], config: dict[str, Any], dtype: torch.dtype
-):
+def get_ltx2_temporal_latent_upsampler_config(version: str):
+    if version != "2.5":
+        raise ValueError(f"Unsupported version: {version}")
+    # The temporal x2 upsampler is narrower than its spatial sibling and pixel-shuffles along time only.
+    return {
+        "in_channels": 128,
+        "mid_channels": 512,
+        "num_blocks_per_stage": 4,
+        "dims": 3,
+        "spatial_upsample": False,
+        "temporal_upsample": True,
+    }
+
+
+def convert_ltx2_latent_upsampler(original_state_dict: dict[str, Any], config: dict[str, Any], dtype: torch.dtype):
     with init_empty_weights():
         latent_upsampler = LTX2LatentUpsamplerModel(**config)
 
@@ -1379,6 +1391,12 @@ def none_or_str(value: str):
             "google/gemma-4-E2B-it or google/gemma-4-E4B-it."
         ),
     )
+    parser.add_argument(
+        "--temporal_latent_upsampler_filename",
+        default="ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors",
+        type=none_or_str,
+        help="Temporal x2 latent upsampler filename (LTX-2.5, used by the DFR pipeline's temporal refine rounds)",
+    )
     parser.add_argument(
         "--latent_upsampler_filename",
         default="ltx-2-spatial-upscaler-x2-1.0.safetensors",
@@ -1410,6 +1428,11 @@ def none_or_str(value: str):
     parser.add_argument("--vocoder", action="store_true", help="Whether to convert the vocoder model")
     parser.add_argument("--text_encoder", action="store_true", help="Whether to conver the text encoder")
     parser.add_argument("--latent_upsampler", action="store_true", help="Whether to convert the latent upsampler")
+    parser.add_argument(
+        "--temporal_latent_upsampler",
+        action="store_true",
+        help="Whether to convert the temporal x2 latent upsampler (LTX-2.5)",
+    )
     parser.add_argument(
         "--full_pipeline",
         action="store_true",
@@ -1608,14 +1631,25 @@ def main(args):
             repo_id=args.original_state_dict_repo_id, filename=args.latent_upsampler_filename
         )
         latent_upsampler_config = get_ltx2_spatial_latent_upsampler_config(args.version)
-        latent_upsampler = convert_ltx2_spatial_latent_upsampler(
+        latent_upsampler = convert_ltx2_latent_upsampler(
             original_latent_upsampler_ckpt,
             latent_upsampler_config,
             dtype=vae_dtype,
         )
         if not args.full_pipeline and not args.upsample_pipeline:
             latent_upsampler.save_pretrained(os.path.join(args.output_path, "latent_upsampler"))
 
+    if args.temporal_latent_upsampler:
+        original_temporal_upsampler_ckpt = load_hub_or_local_checkpoint(
+            repo_id=args.original_state_dict_repo_id, filename=args.temporal_latent_upsampler_filename
+        )
+        temporal_latent_upsampler = convert_ltx2_latent_upsampler(
+            original_temporal_upsampler_ckpt,
+            get_ltx2_temporal_latent_upsampler_config(args.version),
+            dtype=vae_dtype,
+        )
+        temporal_latent_upsampler.save_pretrained(os.path.join(args.output_path, "temporal_latent_upsampler"))
+
     if args.full_pipeline:
         is_distilled_ckpt = "distilled" in args.combined_filename
         if is_distilled_ckpt:

src/diffusers/__init__.py

@@ -724,6 +724,7 @@
             "LongCatImageEditPipeline",
             "LongCatImagePipeline",
             "LTX2ConditionPipeline",
+            "LTX2DFRPipeline",
             "LTX2HDRPipeline",
             "LTX2ImageToVideoPipeline",
             "LTX2InContextPipeline",
@@ -1570,6 +1571,7 @@
             LongCatImageEditPipeline,
             LongCatImagePipeline,
             LTX2ConditionPipeline,
+            LTX2DFRPipeline,
             LTX2HDRPipeline,
             LTX2ImageToVideoPipeline,
             LTX2InContextPipeline,

src/diffusers/models/transformers/transformer_ltx2.py

@@ -1115,8 +1115,8 @@ class LTX2VideoTransformer3DModel(
             for a given prompt.
         use_keyframes_abs_pos_embedding (`bool`, defaults to `False`):
             Whether to store a learned `(1, inner_dim)` absolute-position embedding for generated-keyframe tokens
-            (LTX-2.5.1+). When `True`, the weight is kept on the module for load/save; the regular distilled forward
-            path does not consume it until a dedicated keyframes pipeline wires it in.
+            (LTX-2.5). When `True`, tokens selected by `video_keyframes_mask` receive this embedding. The argument is
+            optional; omitting it leaves the distilled forward path unchanged.
     """
 
     _supports_gradient_checkpointing = True
@@ -1388,6 +1388,7 @@ def forward(
         use_cross_timestep: bool = False,
         attention_kwargs: dict[str, Any] | None = None,
         video_self_attention_mask: torch.Tensor | None = None,
+        video_keyframes_mask: torch.Tensor | None = None,
         return_dict: bool = True,
     ) -> torch.Tensor:
         """
@@ -1458,6 +1459,10 @@ def forward(
                 applied to the video self-attention in each transformer block. Values in `[0, 1]` where `1` means full
                 attention and `0` means masked. Used e.g. by the IC-LoRA pipeline to control attention strength between
                 noisy tokens and appended reference tokens. Audio self-attention is not affected.
+            video_keyframes_mask (`torch.Tensor`, *optional*):
+                Optional per-token marker of shape `(batch_size, num_video_tokens, 1)`, non-zero on video tokens whose
+                latent frame encodes a single pixel frame. Those tokens receive `keyframes_abs_pos_embedding`. Ignored
+                when the model was built without `use_keyframes_abs_pos_embedding`.
             return_dict (`bool`, *optional*, defaults to `True`):
                 Whether to return a dict-like structured output of type `AudioVisualModelOutput` or a tuple.
 
@@ -1509,6 +1514,11 @@ def forward(
         hidden_states = self.proj_in(hidden_states)
         audio_hidden_states = self.audio_proj_in(audio_hidden_states)
 
+        # 2.1. Mark tokens whose latent encodes a single pixel frame (causal first frame, generated keyframe slots).
+        if self.config.use_keyframes_abs_pos_embedding and video_keyframes_mask is not None:
+            marker = (video_keyframes_mask > 0).to(dtype=hidden_states.dtype)
+            hidden_states = hidden_states + marker * self.keyframes_abs_pos_embedding.to(dtype=hidden_states.dtype)
+
         # 3. Prepare timestep embeddings and modulation parameters
         timestep_cross_attn_gate_scale_factor = (
             self.config.cross_attn_timestep_scale_multiplier / self.config.timestep_scale_multiplier

src/diffusers/pipelines/__init__.py

@@ -340,6 +340,7 @@
     _import_structure["ltx2"] = [
         "LTX2Pipeline",
         "LTX2ConditionPipeline",
+        "LTX2DFRPipeline",
         "LTX2HDRPipeline",
         "LTX2InContextPipeline",
         "LTX2ImageToVideoPipeline",
@@ -802,6 +803,7 @@
         )
         from .ltx2 import (
             LTX2ConditionPipeline,
+            LTX2DFRPipeline,
             LTX2HDRPipeline,
             LTX2ImageToVideoPipeline,
             LTX2InContextPipeline,

src/diffusers/pipelines/ltx2/__init__.py

@@ -28,6 +28,7 @@
     _import_structure["latent_upsampler"] = ["LTX2LatentUpsamplerModel"]
     _import_structure["pipeline_ltx2"] = ["LTX2Pipeline"]
     _import_structure["pipeline_ltx2_condition"] = ["LTX2ConditionPipeline", "LTX2VideoCondition"]
+    _import_structure["pipeline_ltx2_dfr"] = ["LTX2DFRPipeline"]
     _import_structure["pipeline_ltx2_diffusion_decode"] = ["LTX2VideoDiffusionDecodePipeline"]
     _import_structure["pipeline_ltx2_hdr_lora"] = ["LTX2HDRPipeline", "LTX2HDRReferenceCondition"]
     _import_structure["pipeline_ltx2_ic_lora"] = ["LTX2InContextPipeline", "LTX2ReferenceCondition"]
@@ -49,6 +50,7 @@
         from .latent_upsampler import LTX2LatentUpsamplerModel
         from .pipeline_ltx2 import LTX2Pipeline
         from .pipeline_ltx2_condition import LTX2ConditionPipeline, LTX2VideoCondition
+        from .pipeline_ltx2_dfr import LTX2DFRPipeline
         from .pipeline_ltx2_diffusion_decode import LTX2VideoDiffusionDecodePipeline
         from .pipeline_ltx2_hdr_lora import LTX2HDRPipeline, LTX2HDRReferenceCondition
         from .pipeline_ltx2_ic_lora import LTX2InContextPipeline, LTX2ReferenceCondition