Repository navigation
AnimateDiff text-to-video on SD 1.5, with FreeNoise long clips - #81
Merged
Merged
Conversation
…tediff SD 1.5 runs through the native SD UNet (fixed 8 heads, 768-wide context, 1x1-conv proj_in/out loaded as linears) and the CLIP ViT-L encoder. The AnimateDiff v2/v3 motion modules come from the Apache-2.0 .ckpt files, read with torch::load_state_dict() and converted once to fp16 safetensors. txt2vid_animatediff() takes prompt keyframes (prompt travel). UNet attention now uses fused SDPA, and the spatial transformer group norm uses diffusers' eps 1e-6 (was 1e-5), which takes the fp32 UNet from 1.8e-4 to 7.7e-6 max abs against diffusers. Against diffusers (fp32 CPU): text encoder 3.7e-6, image UNet 7.7e-6, motion UNet v3 6.7e-6 and v2 7.9e-6 max abs; AnimateDiffPipeline end to end at 256 px, 8 frames, 6 steps, same latents: 88.5 dB PSNR.
…ates The v3 and v2 motion modules were trained on 16-frame clips. In one window of 20, 24 or 32 frames both diffusers and diffuseR give noise (512x512x32 on the GPU: R vs diffusers 43.6 dB PSNR, both noise), so txt2vid_animatediff() now attends within overlapping windows (context_length 16, stride 4, pyramid weights) and refuses windows over 16. Frames past the first window start from shuffled copies of earlier noise (shuffle_context). FreeNoise UNet forward vs diffusers, fp32 CPU: 6.8e-6 max abs at 24 frames; at 26 frames 8.3e-6 with diffusers' last-window count, which uses one scalar weight. diffuseR uses per-frame weights there, a proper weighted mean. decode_chunk defaults to 1: the fp32 VAE at 4 frames set a 13.9 GiB peak at 512x512x16 (5.0 GiB at 1, about 2 s slower). On CUDA the pipeline sets the allocator gc gates: R's gc took 78 of 105 s at 512x512x16 under the defaults.
…ong windows The loader sets the allocator gc gates before the first CUDA allocation, so expandable segments take effect (768x768x16 ran out of memory on 5.5 GiB of fragments without), and sizes them from the memory free at load: with rates from the card total, a card shared down to 10.3 GiB never reached the full-gc gate and every size ran out of memory at 9.8 GiB allocated. The motion UNet and the VAE decode loop run a minor gc per layer or frame (diffuseR.block_gc). 512x512x16 at 25 steps: 95 s with 69 s of R gc under torch's default gates, 37 s with 5 s now. Held to 10.3 GiB free by another process: 512x512x16 37 s (9.3 GiB allocated peak), 512x512x32 112 s (9.6 GiB), 768x768x16 106 s (9.9 GiB). context_length above the 16 training frames now warns instead of refusing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AnimateDiff text-to-video on Stable Diffusion 1.5 with the v3 or v2 motion module (Apache-2.0), for the morphing, prompt-travel look without Zeroscope's CC-BY-NC-4.0 weights.
What's in it
animatediff_load_pipeline(),txt2vid_animatediff(),download_sd15(),download_animatediff(). The original.ckptis converted in R withtorch::load_state_dict(), so no Python step.motion_adapter(),load_motion_adapter(),unet_motion()), ported from diffusers (Apache-2.0).num_attention_heads), 1x1 conv projections loaded as linears,sd_pipeline_from_safetensors(model_name = "sd15").prompt = c("1" = ..., "16" = ...)blends the embeddings frame by frame.context_length,context_stride) with pyramid weights, plus shuffled-context noise. The v2 and v3 modules were trained on 16-frame clips. In a single window of 20, 24 or 32 frames, both diffusers and diffuseR produce noise, socontext_lengthabove 16 warns.Verification
Against diffusers on the real weights, fp32 CPU (max abs):
Measured on an RTX 5060 Ti 16 GB, 25 steps, with a second process holding the card to 10.3 GiB free:
Speeds match the free card. The gc changes took 512x512x16 from 95 s, 69 s of it R gc, to 37 s.
Tests:
test_animatediff.R(46 results) plus the existing UNet, UNet safetensors and text encoder tests pass.Not included: v2 camera-motion LoRAs, SDXL motion module.