fix: derive text-encoder alpha_pattern keys from the LoRA module name

[akhil9tiet]

Fixes #14970

In get_peft_kwargs, the text-encoder alpha_pattern key derivation dropped the LoRA module segment (to_k_lora), producing the parent block (...self_attn) instead of the module (...self_attn.k_proj). Since PEFT matches alpha_pattern keys against the end of module names, the key never matched and every text-encoder module fell back to the most common alpha. The derivation now translates the diffusers LoRA weight suffix (.to_{q,k,v,out}_lora, .lora_linear_layer) to the PEFT module name, and drops the text_model. prefix so the key matches whether or not the loader stripped it for flattened text encoders. Adds two regression tests to tests/lora/test_peft_utils.py.

[andre-menutole]

Thanks for the quick fix, @akhil9tiet! I checked this PR (at e1e02ab) against the reproduction from #14970 on diffusers 0.41.0, transformers 5.16.1 and peft 0.20.0:

• CLIPTextModel (flattened, no text_model level): k_proj scaling is now 1.0 (was 0.25), and alpha_pattern is {'encoder.layers.1.self_attn.k_proj': 4.0}.
• CLIPTextModelWithProjection (still has text_model): k_proj scaling is 1.0 as well.
• The modules with the most common alpha keep 0.25 (alpha 1 / rank 4), as expected.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for the PR. It seems to be doing what it's supposed to do. It depends on some LoRA checkpoint convention that I'm not familiar with, so I can't comment on whether it solves a real world issue or not.

Regarding the implementation itself, I have suggestion and a question.

[BenjaminBossan]

it doesn't actually create 4 rank entries. The comprehension {k.replace(".lora_A.", ".lora_B."): 4 for k in peft_state_dict if ".lora_A." in k} filters to the two .lora_A. keys first, so rank_dict ends up with just the two .lora_B. module names. It's a compact way to build a rank_dict whose keys match the state dict's module names, mirroring what the real checkpoint loader passes in.

It feels a bit roundabout to me compared to just spelling out the rank_dict, but it's no blocker for me.