[akleine]

If runnnig LoRAs using --lora-apply-mode auto the automatic decision between modes "immediately" and "at_runtime" works very well, but there is still a minor gap if using command line options like --type q4_0 etc.

This PR should solve the problem by checking the --type option.

Summary

Two examples will show the problem:

This first runs very well in at_runtime mode:

./sd-cli  -m ~/SD_models/sd1/picxReal_10.safetensors_Q4_0.gguf  \
  --lora-model-dir ~/SD_models/loras_sd15 \
  -pL "A cute cat<lora:LCM-lora-sdv1-5:1>" \
  --sampling-method lcm --steps 4 --cfg-scale 

The sceond is similar but will crash because the --lora-apply-mode is is wrong state.
It runs in immediately mode - but should not:

./sd-cli  -m ~/SD_models/sd1/picxReal_10.safetensors \
  --lora-model-dir ~/SD_models/loras_sd15 \
  -p "A cute cat<lora:LCM-lora-sdv1-5:1>" \
  --sampling-method lcm --steps 4 --cfg-scale 1 \
  --type q4_0

At all: using --type q4_0 is the difference.

Related Issue / Discussion

Additional Information

#3  0x00005cc43b03dbd7 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#4  0x00005cc43b051580 in ggml_cuda_mul_mat_cublas(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggmusing command line options like  --type q4_0 l_tensor*) ()
#5  0x00005cc43b0567f2 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#6  0x00005cc43be91ce5 in ggml_backend_graph_compute ()
#7  0x00005cc43aad6bc8 in GGMLRunner::execute_segment(ggml_cgraph*, int) ()
#8  0x00005cc43aada37a in GGMLRunner::execute_graph(ggml_cgraph*, int, bool, std::function<bool ()> const&) ()
#9  0x00005cc43aadbd2f in GGMLRunner::compute(std::function<ggml_cgraph* ()>, int, bool, bool, std::function<bool ()> const&) ()
#10 0x00005cc43aa6fd24 in LoraModel::apply(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ggml_tensor*, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ggml_tensor*> > >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, SDVersion, int, bool) ()
#11 0x00005cc43aa6cda8 in ModelManager::apply_loras_to_params(std::vector<ModelManager::TensorState*, std::allocator<ModelManager::TensorState*> > const&) ()

Checklist

• I have read and confirmed this PR follows the contribution guidelines.

Summary by CodeRabbit

• Bug Fixes
  • Corrected automatic LoRA application selection for quantized weight configurations, so they use runtime application rather than immediate application.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: b9035def-8edf-4c2a-8cab-617b6c7f8df7

📥 Commits

Reviewing files that changed from the base of the PR and between ee7098e and a1ce56f.

📒 Files selected for processing: 1

• src/pipeline/diffusion_engine.cpp

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

configure_weight_loading now detects quantized expected_type values in tensor storage rules. In automatic LoRA mode, this detection selects runtime application alongside the existing conditions.

Changes: Weight Loading Configuration

Layer / File(s)	Summary
Quantized tensor detection and LoRA selection src/pipeline/diffusion_engine.cpp	The tensor-storage scan detects quantized GGML expected_type values. Automatic LoRA mode selects runtime application when such a value is present.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Suggested reviewers: leejet

Merge Risk: ⚪ Minimal · up to a1ce5

Automatic LoRA mode now defers application when an expected type is recognized as quantized. The inspected code revealed no concrete issue requiring resolution before merge.

### Architecture Summary

Architecture risk: 🔵 Low · up to a1ce5

The change affects 1 system.

Changed systems: src

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

• observed — src (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

• observed — Modified behavior in src/pipeline/diffusion_engine.cpp: The tensor-storage scan now records whether any expected_type is a quantized GGML type, in addition to tracking INT8 tensorwise storage; it stops once both conditions are found.
• observed — Modified behavior in src/pipeline/diffusion_engine.cpp: Automatic LoRA mode now selects runtime application when a quantized tensor-type rule is present, even if the existing quantized-weight, constrained-streaming, and row-split conditions are absent.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description check		The description explains the --type q4_0 issue, provides reproduction examples, includes crash context, and completes the checklist. The Summary section retains its placeholder, and no related issue…
Title check		The title clearly and concisely describes the main change: improving automatic --lora-apply-mode decisions for quantized parameters.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains the --type q4_0 issue, provides reproduction examples, includes crash context, and completes the checklist. The Summary section retains its placeholder, and no related issue is linked, but the required information is otherwise clear.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

🔧 Cppcheck (2.22.0)

src/pipeline/diffusion_engine.cpp

Cppcheck timed out; analysis of this file is incomplete

_{Comment @coderabbitai help to get the list of available commands.}

[leejet]

Could we check each tensor’s effective type (expected_type when set, otherwise type) instead of only the global wtype?

--tensor-type-rules takes precedence over --type, but get_wtype_stat() only counts the original storage types. With a floating-point model:

• --tensor-type-rules 'model.diffusion_model.*=q4_0' still selects immediately, leaving the same class of issue unresolved.
• --type q4_0 --tensor-type-rules '.*=f16' now selects at_runtime, even though the resulting weights are not quantized.

Checking the effective types would cover both cases while preserving the existing INT8 tensorwise handling.

[akleine]

@leejet Thank you for your review!
Actually, I had never used the --tensor-type-rules option - as you might have guessed.

So here comes an update that addresses the mentioned issues.
Perhaps some parts can be shortened a bit more. (Please feel free to edit some more.)
Once again thank you for your work ❤️

[leejet]

Thank you for your contribution.