feat(minimax-h3): portable LoRA keys, pruned and fp8 checkpoint - #41
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MiniMax H3: portable LoRAs, smaller checkpoints, and a fuse that stops eating host RAM
Three users reported that a LoRA trained here has no effect, both in Inline and in other tools.
This fixes that, plus three further defects found while investigating it.
The actual bug
H3's checkpoint stores attention as one fused
blocks.N.attn.qkv_proj. Our loader splits it intoto_q/to_k/to_vwhile streaming, so the adapter PEFT produces is keyed to names no othertool has ever seen. Other tools matched nothing and skipped every layer silently, which looks
exactly like a LoRA that loaded and did nothing.
What changed
LoRA interop, both directions (
models/minimaxh3/lora_keys.py)One bidirectional map built on the existing
keys.pyplan, so it cannot drift from the checkpointloader. Training now writes the published key names, and third-party H3 LoRAs load here.
Two things make it exact rather than a rename table:
r, because each has its ownA. StackedAplus ablock-diagonal
Bgives an identical delta at rank3r, and the alpha triples with it soalpha / rankis unchanged. Missing that last part divides the adapter by three, silently.SwiGLUreads[value; gate]and thereference stores the other order.
The fused QKV row order differs by publisher and cannot be measured from an adapter, which has
no base weights to compare against. Import defaults to the Comfy-Org layout and says so in the log
rather than guessing.
.alphais now written (training/trainer.py)PEFT trains with a scale of
alpha / rankand saves the factors raw, so an adapter without an alphafused at 1.0. Correct only while
alpha == rank, which is the default and is why it went unnoticed.Affects every architecture.
Pruned and fp8 checkpoints load (
minimaxh3/adaln.py,keys.py,load.py,requirements.py)66.3GB down to 21.0GB for the same model. The pruned builds ship no timestep path at all, only
adaln_t_table [1025, 8], so the branch cannot be rebuilt as a basis applied to asilu(temb)wecompute.
adaln.tabulaterebuilds the modules around the table instead, interpolating off-gridtimesteps. fp8 is a scalar
weight_scaleper weight, dequantised while streaming.int8_convrotis still refused: its weights are stored rotated and that is a transform we cannotinvert. An unrecognised quantisation is refused rather than guessed at.
The fuse no longer materialises the whole delta (
models/lora.py)Other fixes
[96768, 8]projection by a full-width basis, the structural sibling of never re-quantising aprequantized file.
lost its AdaLN branch and an fp8 file stores half the bytes it will occupy, so scaling the on-disk
number under-sized both by up to 3x, and under-sizing kills the host instead of raising.
from and saves nothing in VRAM anyway.
Verification
0.000e+00