Add VideoDeltaNet-H3 Model

Model/Pipeline/Scheduler description

VideoDeltaNet (VDN-H3) is a hybrid-attention variant of MiniMax H3 designed for faster video generation with near-lossless quality. It replaces dense video-video attention with a combination of local sliding-window softmax attention and long-range linear attention, while keeping the original H3 backbone largely unchanged.

All weights and training and inference code are available.

Open source status

The model implementation is available.
The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Weight: https://huggingface.co/OpenVDN/vdn-minimax-h3
Code: https://github.com/OpenVDN/vdn-minimax-h3
I'm the author :)

Thanks for the request!

How new model support works in Diffusers

We're a small team, and our review queue shouldn't be what decides whether a model is usable in Diffusers. With Modular Diffusers, a pipeline can live as remote code in any Hub repo and load straight from there with from_pretrained.

🛠️ Want to bring this model to Diffusers?

Please start with a Hub repo — you don't need anything from us to do that, and people can use it immediately. From there we decide how to support it: we might work with the authors, upstream an existing community version, or just point people at the one on the Hub. The pipelines we integrate are usually the ones people are already running.

Tag @asomoza when you have something to share — we'll give feedback on the implementation, help get it in front of people, and add the ones we like to our hand-picked Modular Pipelines collection. Tell us where you hit friction along the way, too: confusing APIs, missing docs, bugs. That feedback is worth as much to us as the pipeline.

👋 Are you an author of the model? We'd love to hear from you — comment here and we'll help you pick the path that fits.

📚 Quickstart · Building custom blocks — template repo, and how to publish to the Hub · Modular Pipelines and Custom Blocks — examples to crib from

This is an automated message.

@asomoza

Thanks for the request!

How new model support works in Diffusers

We're a small team, and our review queue shouldn't be what decides whether a model is usable in Diffusers. With Modular Diffusers, a pipeline can live as remote code in any Hub repo and load straight from there with from_pretrained.

🛠️ Want to bring this model to Diffusers?

Please start with a Hub repo — you don't need anything from us to do that, and people can use it immediately. From there we decide how to support it: we might work with the authors, upstream an existing community version, or just point people at the one on the Hub. The pipelines we integrate are usually the ones people are already running.

Tag @asomoza when you have something to share — we'll give feedback on the implementation, help get it in front of people, and add the ones we like to our hand-picked Modular Pipelines collection. Tell us where you hit friction along the way, too: confusing APIs, missing docs, bugs. That feedback is worth as much to us as the pipeline.

👋 Are you an author of the model? We'd love to hear from you — comment here and we'll help you pick the path that fits.

📚 Quickstart · Building custom blocks — template repo, and how to publish to the Hub · Modular Pipelines and Custom Blocks — examples to crib from

This is an automated message.

@asomoza
The implementation is on the Hub and loads as a Modular Diffusers pipeline, no fork and no patched diffusers:
https://huggingface.co/OpenVDN/vdn-minimax-h3

It reuses MiniMaxH3Blocks unchanged. The only thing we ship is the transformer, a hybrid-attention variant of MiniMaxH3Transformer3DModel. Every other component still points at MiniMaxAI/MiniMax-H3. Can be used through:

pipe = ModularPipeline.from_pretrained("OpenVDN/vdn-minimax-h3", workflow="t2va")
pipe.load_components(trust_remote_code=True, torch_dtype=torch.bfloat16)

We would like to see if we can merge it into Diffusers repo. Thanks so much!

@asomoza

cd @asomoza
can you add to our collection?

By the way, while trying to implement offloading for VDN-H3, we found this PR really helpful. However, I noticed that it’s being closed.

Would you consider fixing the bug, or would you prefer that I submit a PR?

HI, seems you didn't put the link to the PR, but if it helps, I got it to run on a 40GB GPU with this:

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained(
    "OpenVDN/vdn-minimax-h3", workflow="t2va", components_manager=manager, collection="vdn"
)
pipe.load_components(names="transformer", trust_remote_code=True, dtype=torch.bfloat16, fp8={"transformer": True})
pipe.load_components(trust_remote_code=True, dtype=torch.bfloat16)

offload = {
    "onload_device": torch.device("cuda"),
    "offload_device": torch.device("cpu"),
    "use_stream": True,
    "low_cpu_mem_usage": True,
}

pipe.transformer.enable_group_offload(offload_type="block_level", num_blocks_per_group=1, **offload)
apply_group_offloading(pipe.text_encoder.model, offload_type="leaf_level", **offload)
pipe.vae.to("cuda")
pipe.audio_vae.to("cuda")

out = pipe(
    prompt=prompt,
    num_frames=192,
    num_inference_steps=9,
    generator=torch.Generator().manual_seed(0),
    output=["videos", "audio", "sampling_rate"],
)

#13238
The link to the PR ↑.

The problem I encountered is that when I quantize the linear layers into FP8 and register the weights as buffers, it can not be offloaded correctly when trying to stream the offloading process. By adding the fix above, it works. So maybe adding that fix back will be helpful!

@haochengxi

@haochengxi what library/method are you using to quantize the weights? Is it custom? I've asked the author of #13238 to reconsider opening the PR in the meantime.

It's a custom quantized method (FP8 Linear Quantization). I think that the PR's fix is quite general, not limited to custom FP8 Quantization. Thanks so much!

@haochengxi

@haochengxi btw, from what i can tell you allow either your custom fp8 quant or original bf16. but if user passes quantization_config, it is ignored. afaik, there is no reason to skip user-selected quantization and force bf16?

@haochengxi btw, from what i can tell you allow either your custom fp8 quant or original bf16. but if user passes quantization_config, it is ignored. afaik, there is no reason to skip user-selected quantization and force bf16?

Thanks let me check for more details about what's the best practice.

@vladmandic

@vladmandic The quantization is torchao now. The weights are ordinary parameters,
so apply_group_offloading(..., use_stream=True) on stock diffusers streams them as they
are, and we no longer need the fix from #13238 on our side. (but still, I think that PR makes sense)

For quantization config, it needs to be more cautious: the loader
quantizes each weight as it loads, and this component merges LoRA adapters into
Linear.weight after loading, which a quantized weight cannot take (that is where a
passed-through config failed before, rather than being dropped). from_pretrained now
catches it, loads bf16, merges, then applies it. It accepts a TorchAoConfig and
raises on any other backend; fp8=True is just a preset TorchAoConfig.

from diffusers import TorchAoConfig
from torchao.quantization import Float8DynamicActivationFloat8WeightConfig, PerRow

pipe.load_components(
    trust_remote_code=True, torch_dtype=torch.bfloat16,
    quantization_config={"transformer": TorchAoConfig(
        Float8DynamicActivationFloat8WeightConfig(granularity=PerRow()))},
)

@asomoza on memory: with the transformer offloaded per block (offload_type="block_level",
num_blocks_per_group=1, use_stream=True) and the other components offloaded, the
full 345 frames now peak at 22 GB in bf16 and 20 GB in fp8, so the 14 s clip runs on a
24 GB card.

@asomoza

Hi @asomoza, thanks for testing this!

Just following up on upstreaming VDN-H3. Since the Modular Pipeline is working now and most components are reused directly from MiniMax H3, would it make sense to upstream the VDN transformer into Diffusers? I'm happy to open a PR and add the implementation/tests/docs if that's a direction you'd be open to.

Also, I don't think the Hub repo has made it into the Modular Pipelines collection yet. Happy to make any changes needed on our side.

@haochengxi

Hi @haochengxi it is here: https://huggingface.co/collections/diffusers-modular/community-modular-diffusers-blocks but we have a small list for now and still deciding where we will put it so it is more visible, this will take some time.

About adding the VDN transformer, we usually do it only for models that are very popular, but lets wait for @yiyixuxu decision about this.

Navigation Menu

Repository navigation

Add VideoDeltaNet-H3 Model #14700

Model/Pipeline/Scheduler description

Open source status

Provide useful links for the implementation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add VideoDeltaNet-H3 Model #14700

Description

Model/Pipeline/Scheduler description

Open source status

Provide useful links for the implementation

Activity

github-actions commented on Sep 4, 2026

haochengxi commented on Sep 10, 2026

yiyixuxu commented on Sep 10, 2026

haochengxi commented on Sep 11, 2026

asomoza commented on Sep 12, 2026

haochengxi commented on Sep 12, 2026

DN6 commented on Sep 14, 2026

haochengxi commented on Sep 14, 2026

vladmandic commented on Sep 14, 2026

haochengxi commented on Sep 15, 2026

haochengxi commented on Sep 15, 2026

haochengxi commented on Sep 17, 2026

asomoza commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions