Add split GPU text encoder cache - #9310
Closed
Jacid23 wants to merge 1 commit into
Closed
Conversation
github-actions
Bot
added
api
python
Jacid23
marked this pull request as ready for review
June 28, 2026 03:04
Jacid23
requested review from
JPPhoto,
Pfannkuchensack,
blessedcoolant,
dunkeroni and
lstein
as code owners
June 28, 2026 03:04
Collaborator
|
This is a great idea. Heads up that this is a generic multi-GPU PR coming down the pike (#5997) and this will need some adaptation to work with that scheme. I'll be working on an integration. |
This was referenced Jun 28, 2026
lstein
added a commit
to lstein/InvokeAI
that referenced
this pull request
Jun 29, 2026
Adds `offload_text_encoders_to_idle_gpus` (default on): when more than one generation device is configured and a GPU is idle, a session's text/prompt encoder runs on the idle GPU instead of the one running its denoise pipeline. This avoids evicting the denoise model from VRAM to make room for the encoder, and lets a cached encoder be reused across generations. Under full load (no idle GPU) behavior is unchanged. Mechanism: - New GENERATION_DEVICE_POOL arbiter (backend/util/device_pool.py) with a per-device exclusive-use lock. A native session blocking-acquires its own device's lock for the whole run; an encoder node try-borrows an idle device's lock for the duration of the node. This makes a borrowed encoder and a native session mutually exclusive on a GPU -- preventing the shared-encoder corruption that produced garbled images -- and is deadlock-free (borrows are non-blocking; a session only ever blocks on its own device). - DefaultSessionRunner re-pins the worker thread to the borrowed device for the whole encoder node; conditioning is stored on the CPU and the denoiser picks it up on its own GPU afterward. - Nodes opt in via @invocation(idle_gpu_offloadable=True), mirroring the existing `bottleneck` ClassVar marker. Applied to the text/prompt encoder nodes (compel + sdxl/refiner, flux, sd3, qwen-image, anima, cogview4, flux2 klein, z-image, flux_redux). Inspired by invoke-ai#9310; supersedes it. Tests: device-pool lock semantics, two concurrency regression tests asserting a session and a borrow never use a GPU at the same time, the runner offload context-manager behavior, and a marker-wiring check. Docs: invokeai-yaml.mdx (config setting) and creating-nodes.mdx (how to support the feature in a node). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
|
Because of pending #9263 , this PR will be in conflict and can't be merged. However, the idea has been folded into a pending PR in my personal repository that will be posted here after 9263 goes in. It is in lstein#137 if you'd like to take a look. I will give full credit to @Jacid23 for the concept and initial implementation. |
5 tasks
Collaborator
|
Superseded by #9263 |
lstein
added a commit
that referenced
this pull request
Jul 30, 2026
* feat(app): parallel multi-GPU session execution
Run one generation session per configured GPU concurrently, with a tiled
progress preview. Multi-user isolation is unchanged. Backed by five seams:
- Per-thread device context (TorchDevice.set/get/clear_session_device);
choose_torch_device() consults it first, so all device-selecting call sites
resolve to the calling worker's GPU with no per-node changes.
- Per-device model caches: build_model_manager builds one ModelCache per
generation device; ModelLoadService.ram_cache resolves by current thread
device; ram_caches fans out clear/drop/shutdown.
- Atomic concurrent dequeue: a dequeue lock makes select+claim atomic so
concurrent workers never claim the same item (works on FIFO; round-robin
from #9086 slots in later).
- Worker pool: one _SessionWorker per device, each pinning torch.cuda.set_device
and its session device, with its own runner and cancel event; cancellation
routes via an {item_id -> worker} lookup. Single-device installs keep the
exact legacy single-worker behavior. Profiling disabled when >1 worker.
- New config `generation_devices`; unset = legacy single-worker mode.
Frontend: the canvas staging area already tiles per queue item; the main
ImageViewer now tracks progress per session and renders a tile grid
(ProgressImageTiles) when more than one session is active.
Also adds a lock to ObjectSerializerForwardCache for concurrent access.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(tests): restore global device after multi-GPU cache routing test
test_model_load_device_routing mutated the process-wide get_config()
singleton (device = "cuda:0") to exercise the per-thread cache routing,
but never restored it. The leaked CUDA device was then picked up by a
later test (test_model_load::test_loading) via choose_torch_device(),
which crashed with "Torch not compiled with CUDA enabled" on the
CUDA-less CI runner. Add an autouse fixture to save/restore device and
clear any pinned session device.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(ui): regenerate openapi schema and frontend types for generation_devices
Regenerate openapi.json (make frontend-openapi) and the frontend
schema.ts types (make frontend-typegen) so they include the new
generation_devices config field, fixing the openapi-checks and
typegen-checks CI jobs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): regenerate openapi.json with uv to match CI generator
`make frontend-openapi` used a bare `python` from a different environment
that emitted the CacheStats @dataclass docstring as a schema description.
CI generates the schema via `uv run`, which does not, so openapi-checks
failed on the diff. Regenerate with the uv-locked environment to drop the
stray description while keeping the generation_devices field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(model-manager): serialize model construction against VRAM moves to prevent meta-device corruption
Parallel multi-GPU session workers could intermittently crash with "unrecognized
device meta" (denoise) or "Cannot copy out of meta tensor; no data!" (l2i), because
model loading relies on process-global, non-thread-safe monkey-patches.
accelerate.init_empty_weights() (used directly by the loaders and implicitly by
diffusers' default low_cpu_mem_usage=True in from_pretrained) swaps
torch.nn.Module.register_parameter globally for the duration of a load, routing every
newly-registered parameter to the meta device. The model cache's VRAM load/unload runs
nn.Module.load_state_dict(assign=True), whose assign path does setattr -> __setattr__ ->
register_parameter. When one worker's VRAM move overlapped another worker's from_pretrained,
the move's real weights got hijacked onto meta and blew up on the next .to(device).
Introduce MODEL_LOAD_LOCK, a write-preferring readers-writer lock:
- write lock = model construction (_load_and_cache, load_model_from_path), exclusive.
- read lock = VRAM load/unload (ModelCache.lock(), repair_required_tensors_on_device).
VRAM transfers across GPUs still overlap each other; they only block while a construction
holds the write lock. The lock is always acquired before any per-cache lock to keep a
consistent order and avoid an AB-BA deadlock with the writer's make_room/put.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): fix outpainting crash caused by model download collisions
* fix(backend): make DiskImageFileStorage thread-safe for parallel sessions
Image.open() is lazy: it reads the header but defers pixel decoding (and
holds the file handle open) until the first .load()/.copy()/.convert(). The
opened object was cached and the same object handed to every caller, so in
multi-GPU parallel mode two session-processor worker threads could call
.copy() on it concurrently and race on the shared file handle and decoder
state. This surfaced as "broken data stream when reading image file" and
"AssertionError: self.png is not None" during inpainting with batch >1.
Force the decode (image.load()) before the object enters the cache so the
cached object is safe for concurrent reads, and guard the cache structures
(__cache / __cache_ids) with a lock since they are now mutated from multiple
threads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(ui): stack per-session progress bars during parallel generation
The generation progress bars (under the Invoke button and the Viewer tab)
both read a single global $lastProgressEvent atom, which every session
overwrites. With parallel multi-GPU sessions this made the bar jump back
and forth between sessions.
Track progress per queue item id and render one bar per in-flight session,
stacked vertically, each removed as its session reaches a terminal state.
- stores.ts: add $progressEvents (map keyed by item_id),
$activeProgressEvents (sorted), and set/clear helpers.
- setEventListeners.tsx: populate per-item progress on invocation_progress;
clear per item on terminal status; clear all on connect/disconnect/queue
cleared.
- ProgressBar.tsx: render a vertical stack of bars (one per active session)
with a single-bar fallback for the idle / model-loading window; add
containerProps so dockview tabs can position the stack.
- Dockview tab call sites: move positioning into containerProps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): make $progressEvents module-local to satisfy knip
$progressEvents is only referenced within stores.ts (via the
$activeProgressEvents computed and the set/clear helpers), so exporting
it tripped knip's unused-exports check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): cap stacked tab progress bars to fit below the tab label
With 4 GPUs the stacked per-session progress bars grew past the bottom
strip of the dockview tab and overlapped the "Viewer" label.
Add a fitHeightPx prop: in fit mode the stack is capped to the available
strip (10px below the ~40px tab's centered label) and the bars flex to
share it, shrinking below their natural height only once they no longer
fit. With 1-2 sessions the bars keep their familiar thin height; with 3+
they scale down to stay within the strip. The sidebar bar is unaffected
and continues to stack at natural height (it has the vertical room).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(config): support "auto" generation_devices to use all GPUs by default
generation_devices now accepts "auto" (the new default), which expands to
every visible CUDA device — so multi-GPU parallel generation works out of
the box without manually listing devices. On GPU-less systems "auto"
resolves to the single cpu/mps device, preserving serial behavior.
- config_default.py: type is now Union[Literal["auto"], list[str]],
default "auto"; validator accepts "auto" or a list of device strings.
- devices.py: add TorchDevice.get_generation_devices(), the single resolver
that expands "auto", normalizes, and deduplicates.
- session_processor / model_manager: both consumers use the resolver
instead of iterating the raw config value (which would have iterated the
characters of the "auto" string).
- Regenerated docs/src/generated/settings.json.
- Tests for the resolver (auto-with/without-CUDA, dedup, empty).
An explicit single-device list (e.g. [cuda:0]) or an empty list opts out
of parallelism.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(frontend): typegen+openapi
* docs(multi-gpu): add configuration information
* chore(frontend): typegen + openapi again
* feat(settings): add Generation Devices selector to Settings dialog
Add a badges UI in the Generation section of the Settings dialog for
choosing which devices `generation_devices` should use, modeled on the
Log Namespaces toggle UI.
Backend:
- New `GET /api/v1/app/generation_device_options` endpoint listing the
selectable devices (cuda:N with GPU names, or the sole mps/cpu fallback).
- Add `generation_devices` to the runtime-config update allowlist with
validation rejecting invalid device strings and explicit nulls.
Frontend:
- New SettingsGenerationDevices component with active/inactive badges.
"Auto (all GPUs)" is exclusive; removing the last explicit device
reverts to auto. Admin/multiuser gated; notes restart requirement.
- Wire into the Generation section; regenerate schema; add en strings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(settings): boldface the restart notice on Generation Devices
Split the restart sentence into its own string and render it bold so
users notice that device changes require restarting InvokeAI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(settings): show GPU name in Generation Devices badges
Render device badges as "cuda:0 (RTX 3090 #1)" so identical cards can be
told apart. Strips the "NVIDIA GeForce" vendor prefix and adds a 1-based
"#N" suffix only when multiple cards share a name. The full device name
remains available as the badge tooltip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(frontend): openapi
* feat(multi-gpu): surface per-session GPU number in logs and UI
Help users track which CUDA device is processing each session:
- Model-load log: "Loaded model ... onto cuda device #N in ..s"
- Denoise progress bars: "Denoising (#N)" across all architectures
(SD1.5/SDXL, FLUX, FLUX2, Z-Image, Anima, SD3, CogView4)
- Progress preview circle: GPU number centered in the ring, via a new
`device` field on InvocationProgressEvent (resolved from the worker's
thread-local session device)
- Session Queue: new "GPU #" column between STATUS and TIME, backed by a
`device` column on session_queue (migration_32) recorded when a worker
claims an item
Adds TorchDevice.get_session_device_label()/get_session_device_index()
helpers and a frontend getCudaDeviceIndex() parser (with tests). Shows the
number on CUDA only; CPU/MPS show nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(multi-gpu): show per-device names in startup log and progress circles
- Startup log lists each generation device with its GPU number and id,
e.g. "Using torch device: [AMD Radeon PRO W7900 #1 (cuda:0), ...]".
Single-device setups keep the bare device name.
- Canvas progress circles now show the CUDA device index in the center,
matching the viewer panel.
- Progress-circle tooltips show the device name and number on hover.
- Both are hidden when only a single GPU is available.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(model-cache): share one CPU copy of model weights across per-GPU caches
In multi-GPU mode the model manager builds one ModelCache per generation device,
each with storage_device="cpu" and its own RAM-resident copy of every model. A model
loaded on N GPUs therefore occupied N copies in RAM, and each cache sized itself
against max_cache_ram_gb independently, so RAM use during the text/reference-image
encoding phases skyrocketed and the system swapped — worst when two images rendered
at once.
This deduplicates the CPU-resident weights and makes RAM accounting global.
- SharedCpuWeightsStore: process-/manager-global, refcounted store of one canonical
CPU state_dict per model key. The first device to load a key registers its weights;
subsequent devices adopt the canonical tensors and re-point their module's params at
them (load_state_dict(assign=True)), freeing the duplicate. Weights live once in RAM
regardless of GPU count; freed only when the last device releases. Per-device modules
are kept (params are device-shuffled in place, so two GPUs need two modules), but
their CPU-resident params alias the shared tensors.
- RamBudget: single system-wide RAM authority. Splits RAM into shared (counted once via
the store) and non-shared (per-instance). ModelCache eviction now runs against the
global, deduplicated total and re-checks availability each iteration, since evicting a
model another device still holds frees no RAM. build_model_manager wires one store +
one budget into all device caches; the cap is max_cache_ram_gb as a true system-wide
limit, else the sum of per-cache heuristics. Passing ram_budget=None preserves the
prior local accounting.
- LoRA/patch safety: direct LoRA patching did an in-place copy_ on the weight, which
would corrupt the now-shared canonical tensor (and taint keep_ram_copy even with one
GPU) when patching a CPU-resident weight. Switched to an out-of-place add (memory-
equivalent) so the canonical tensor is never mutated; fixed the FluxControlLoRA
expansion path to target the module's live parameter. Sidecar patching and
FreeU/Seamless (which patch forward methods) were already safe.
Validated on 2x AMD W7900 / ROCm: correct inference on both GPUs from one shared copy
(full + partial load + Q8_0 GGUF quantized), concurrent load/unload without corruption,
and LoRA isolation across devices. ~40 new tests; existing suites unchanged.
Adds scripts/multigpu_ram_driver.py to drive concurrent dual-GPU generations via the
queue API and measure peak RSS / leak drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(session-queue): cancel all in-progress items in bulk-cancel APIs (multi-GPU)
With one session-processor worker per device, multiple queue items can be in_progress
at once. cancel_by_batch_ids(), cancel_by_destination() and cancel_by_queue_id() excluded
in_progress rows from their bulk UPDATE and then canceled only the single get_current()
item (LIMIT 1), so on multi-GPU the other running items kept consuming a GPU and could
still produce output after the user requested cancellation.
Each running item must be canceled via _set_queue_item_status(), which emits the
QueueItemStatusChangedEvent that the processor maps to the worker running that item_id and
uses to set its cancel event. Add _cancel_in_progress_matching() to cancel every in-progress
item matching the same filter (with user-id scoping preserved) and call it from all three
bulk-cancel methods. The returned `canceled` count now includes canceled in-progress items.
Adds regression tests that dequeue two items onto separate devices and assert every bulk
cancel API moves all matching in_progress items to canceled and emits a cancel event for
each (and that user-scoped cancel leaves another user's in-progress item running).
Reported by JPPhoto in review of #9263.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(multi-gpu): address review findings (cancel race, bulk delete, device guards, refcount leak)
Fixes from the code review of PR #9263:
- Cancellation could be silently lost around dequeue: the per-iteration
worker.cancel_event.clear() ran AFTER dequeue + gc.collect() + logging, so a cancel
arriving in that window was set by the status handler and then wiped. Move the clear to
before dequeue, and after claiming an item re-check (cancel_event + a fresh DB status read
via _is_queue_item_terminal) and skip running if it is already terminal, closing both race
windows. The runner's stale queue_item.status check could not catch this.
- delete_by_destination only stopped one in-progress item (get_current) before deleting all
matching rows, leaving other GPU workers running (and then failing to update a deleted row).
Cancel every matching in-progress item via _cancel_in_progress_matching first.
- generation_devices validation: a bare non-"auto" string (e.g. "cuda:0") was iterated
character-by-character; an empty list silently fell back to one device. Reject both with a
clear message.
- get_generation_devices now fails fast on a CUDA device that does not exist (index past
device_count, or CUDA unavailable) instead of starting a worker that errors cryptically at
first allocation.
- Shared-weights wrappers: if the canonical re-point (load_state_dict assign=True) threw after
acquire(), the reference was leaked (the wrapper never entered the cache). Compute size
metadata first, make acquire the last step, and release on failure.
Adds tests for each: post-dequeue terminal guard, delete_by_destination cancellation,
generation_devices validation, absent-device rejection, and acquire-released-on-repoint-failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ci): ruff format + make CPU-incompatible device test mock CUDA
- Apply ruff 0.11.2 formatting to the files flagged by `ruff format --check`.
- The new fail-fast guard in get_generation_devices() (reject a CUDA device that
doesn't exist) made the pre-existing test_get_generation_devices_explicit_list_is_deduplicated
fail on CPU-only CI runners, since it passes a cuda list with no CUDA present. Mock
torch.cuda.is_available/device_count in that test (matching the existing pattern in this
file) so it validates dedup on any runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(multi-gpu): stop RAM blowup/swapping during concurrent generations
Three RAM fixes for multi-GPU (and one that helps single-GPU too), addressing
transient spikes to ~100% RAM and swapping during text-encode/transformer loads:
1. Cap the global RAM-cache budget at a safe fraction of system RAM. When
max_cache_ram_gb is unset, the budget was the *sum* of the per-device cache
heuristics, so N GPUs each claiming ~50% of RAM summed to ~N*50% and starved
the OS. Now clamp the sum to ModelCache.calc_system_ram_headroom_bytes()
(50% of RAM - 2GB baseline, floored at 4GB). Promote the sizing magic numbers
to named constants shared by the per-device heuristic and the global cap.
2. Adopt already-resident CPU weights across devices at load time. When a second
device loads a model another device already holds, deep-copy a registered
meta-weight structural clone and assign the shared canonical weights, instead
of re-reading the model from disk and materializing a full transient second
copy. Loader-agnostic (one mechanism in ModelLoader, no per-loader code):
works for diffusers, single-file checkpoint, GGUF and transformers models,
and preserves registered hooks (e.g. fp8 layerwise-cast). Best-effort with a
meta-tensor self-check and fallback to a normal disk load on any failure.
Skipped on single-device installs.
3. Dequantize FLUX.2 FP8 checkpoints straight to bf16. _dequantize_fp8_weights
materialized the whole model in float32 (~36GB for 9B) before a later cast to
bf16; now the multiply is done in float32 but stored bf16 per-weight, so the
model is never held in float32. Numerically identical; halves the cold-load
transient (helps single-GPU too).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(qwen-image): reserve VAE working memory so decode/encode don't OOM
The Qwen Image VAE encode/decode invocations called model_on_device() without a
working-memory estimate, unlike every other VAE family (SD/SDXL/SD3/CogView4/FLUX).
So the model cache reserved only its small default working memory, never offloaded
a large resident transformer (the VAE weights themselves are tiny), and the VAE's
forward-pass activations then OOM'd VRAM — e.g. a ~40GB Qwen Image Edit transformer
left ~1GB free while decode needed ~5GB. Reproduces single-GPU; unrelated to the
multi-GPU RAM work.
Add estimate_vae_working_memory_qwen_image() (same per-output-pixel scaling as the
other estimators, handling the 5D Qwen latents) and pass it from both the i2l
(encode, used for reference images in Image Edit) and l2i (decode) nodes, so the
cache offloads the transformer before the VAE runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flux2): tile reference-image VAE encode to avoid VRAM OOM
The FLUX.2 VAE encoder's mid-block self-attention scales quadratically with the
input's spatial size, and on ROCm scaled_dot_product_attention falls back to a
materialized attention matrix. Encoding a reference image (kontext) at full size
therefore allocated ~15GB in a single attention call at 1024px — and hundreds of
GB at the 2024px reference cap — OOMing VRAM regardless of how much other model
memory was freed.
Tile the reference-image encode to bound per-tile attention. The VAE's default
tile size equals its sample_size (1024), whose per-tile attention still OOMs, so
force a 512px tile (with a matching latent tile size derived from the config).
Save/restore the VAE's tiling config since it is a shared, cached instance, so the
final image decode does not inherit these settings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(multi-gpu): query execution device for VRAM-in-use accounting
ModelCache._get_vram_in_use() called torch.cuda.memory_allocated() with no device
argument, while _get_vram_available() reads memory_allocated(execution_device).
The formula relies on those two canceling. In multi-GPU mode each worker calls
torch.cuda.set_device for its own GPU, so the process-current device flips between
workers; the no-argument call can then read a different (e.g. idle) GPU's
allocation, breaking the cancellation and inflating "available" VRAM toward the
card total. The cache then believes there is room and never offloads, so VRAM
offloading effectively ignores device_working_mem_gb in multi-GPU. Single-GPU was
unaffected (current device always equals the execution device).
Query self._execution_device in both _get_vram_in_use() and the cache-state debug
log. Add a regression test asserting the per-cache execution device is used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(qwen-image): calibrate VAE working-memory estimate to the 3D-conv decode peak
The Qwen Image VAE is a 3D-conv (video) VAE whose decode allocates large conv3d
feature maps. A ~1MP decode was measured to peak at ~17 GiB of VRAM — far above
what the generic 2200/1100 SD/FLUX constants reserved (~4.6 GiB), so the cache
concluded the decode "fit" alongside the resident 20GB transformer + 15GB text
encoder, never offloaded them, and OOMed. The offload only frees ~(working_mem -
free) bytes, so the reservation must both cover the real peak and be large enough
to trigger the offload of models the decode doesn't need.
Raise the Qwen decode/encode constants (13000/6500) to match the measured peak.
It's linear in output pixels, so it over-reserves past ~1.5MP (where the decode
can exceed the card even after offloading) — that case is covered by
force_tiled_decode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(qwen-image): honor force_tiled_decode in the l2i node
The Qwen Image latents-to-image node hardcoded vae.disable_tiling(), ignoring the
global force_tiled_decode setting that the SD/SDXL l2i node honors. Wire it up the
same way so users can opt into tiled VAE decode for very large outputs that exceed
VRAM even after the transformer/text encoder are offloaded. Off by default, so
normal-size decodes are unchanged (full-frame, no tile blending).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): stop progress disk flashing during indeterminate phases
The preview-panel progress circle re-renders on every InvocationProgressEvent. The
parent passes a fresh progressEvent object each event, so the CircularProgress
re-rendered constantly; during the indeterminate phases (everything except
denoising) that restarted its CSS spin animation each time, which looked like the
disk flashing. (Determinate denoising was unaffected because the value genuinely
changes per step.)
Split the circle into a memoized, ref-forwarding subcomponent keyed on its visual
props (isIndeterminate, value, device label) so message-only updates no longer
re-render it and the spin animation stays continuous. The Tooltip still anchors to
it via the forwarded ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(multi-gpu): offload text encoders to idle GPUs
Adds `offload_text_encoders_to_idle_gpus` (default on): when more than one
generation device is configured and a GPU is idle, a session's text/prompt
encoder runs on the idle GPU instead of the one running its denoise pipeline.
This avoids evicting the denoise model from VRAM to make room for the encoder,
and lets a cached encoder be reused across generations. Under full load (no
idle GPU) behavior is unchanged.
Mechanism:
- New GENERATION_DEVICE_POOL arbiter (backend/util/device_pool.py) with a
per-device exclusive-use lock. A native session blocking-acquires its own
device's lock for the whole run; an encoder node try-borrows an idle device's
lock for the duration of the node. This makes a borrowed encoder and a native
session mutually exclusive on a GPU -- preventing the shared-encoder
corruption that produced garbled images -- and is deadlock-free (borrows are
non-blocking; a session only ever blocks on its own device).
- DefaultSessionRunner re-pins the worker thread to the borrowed device for the
whole encoder node; conditioning is stored on the CPU and the denoiser picks
it up on its own GPU afterward.
- Nodes opt in via @invocation(idle_gpu_offloadable=True), mirroring the
existing `bottleneck` ClassVar marker. Applied to the text/prompt encoder
nodes (compel + sdxl/refiner, flux, sd3, qwen-image, anima, cogview4, flux2
klein, z-image, flux_redux).
Inspired by #9310; supersedes it.
Tests: device-pool lock semantics, two concurrency regression tests asserting a
session and a borrow never use a GPU at the same time, the runner offload
context-manager behavior, and a marker-wiring check.
Docs: invokeai-yaml.mdx (config setting) and creating-nodes.mdx (how to support
the feature in a node).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(multi-gpu): adopt GGUF weights across devices to stop RAM spikes
_build_meta_shell built meta placeholders with torch.empty_like, which
GGMLTensor.__torch_dispatch__ rejects (NotImplemented for aten.empty_like).
It threw on the first parameter, hit the silent except, and returned None —
so GGUF models (e.g. a Q8_0 transformer) never registered a shell and the
second GPU re-loaded the full model from disk, stacking a ~20GB transient on
the retained copy and spiking RAM to ~70%.
Fall back to a plain meta placeholder (logical shape/dtype) when empty_like
isn't implemented by a tensor subclass; verified the adopted GGMLTensor shares
the quantized storage, so it's one RAM copy across devices. Peak drops ~66→~46GB.
Log shell-build failures at debug so a future un-adoptable family is diagnosable
instead of silently double-loading.
Also restore log_memory_usage's per-cold-load RAM logging (the capture method
had no callers), slimmed to baseline→transient-peak process RAM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(multi-gpu): tie device #N label to cuda index, not filtered position
The backend device summary computed the disambiguating #N suffix by
enumerating the filtered generation_devices list, so disabling a device
(e.g. cuda:1) renumbered the survivors. The frontend labels over the full
device set, so the two disagreed. Compute the suffix over all available
devices instead, keeping the label stable and consistent with the frontend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(multi-gpu): flash restart reminder when generation devices change
Reword the Generation Devices caption to "Restart InvokeAI for changes to
take effect." and flash that same warning as a toast on every successful
change, so the restart requirement is hard to miss.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(queue): device-affinity dequeue to reduce model reload thrash on multi-GPU
When a GPU worker dequeues, prefer — among the fairness-chosen user's
equal-priority pending items — one whose models are already resident in that
device's cache. Cross-device model reloads cost tens of seconds for large
models; picking a warm item instead cuts thrash when a user queues a mix of
models.
Guardrails (from adversarial review):
- Round-robin user choice and priority tiers are never overridden; the swap
pool is limited to the candidate's user and priority.
- The swap window is capped at AFFINITY_MAX_LOOKAHEAD past the candidate's
item_id, bounding both cold-item deferral and per-dequeue scan cost.
- Explicitly configured session_queue_mode=FIFO opts out of reordering.
- Resident keys are snapshotted before the dequeue lock, and
ModelCache.cached_model_keys() acquires its lock non-blockingly, so a
long-running VRAM transfer can never stall other workers' dequeues.
- Path-keyed cache entries (load_model_from_path) are excluded so a Windows
drive letter can't poison substring scoring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(qwen): restore legacy key remapping for single-file VL encoders under transformers 5.x
The single-file Qwen2.5-VL encoder loader relied on
Qwen2_5_VLForConditionalGeneration._checkpoint_conversion_mapping to translate
ComfyUI's legacy key layout (visual.*, model.layers.*) to the modern one
(model.visual.*, model.language_model.*). transformers 5.x ships that mapping
empty — the conversion moved into from_pretrained's weight-converter machinery,
which our manual load_state_dict path bypasses — so the vision tower was left
on the meta device and loading failed with "Meta tensors remain".
Fall back to the equivalent hardcoded mapping when the class attribute is
empty or absent. Verified against qwen_2.5_vl_7b_fp8_scaled.safetensors:
loads all 8.29B params with no meta tensors remaining.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: address multi-GPU review findings from PR #9263 review
- Shared CPU weights: drop_model() now invalidates the model's canonical
entries in SharedCpuWeightsStore, so a rebuild on another device can
never adopt pre-settings-change weights still aliased by a locked
(stale-marked) entry. release() is identity-checked so a stale
holder's eviction cannot decrement a newly registered canonical.
update_model_record holds MODEL_LOAD_LOCK.write_lock() (off the event
loop) across the multi-cache drop to exclude in-flight loads.
- Runtime config API: generation_devices is now fully validated at the
route boundary — empty lists and unavailable devices (e.g. cuda:99)
return 422 without mutating or persisting config, using the same
TorchDevice resolution as startup.
- Cache stats: /v2/models/stats aggregates per-device caches instead of
reporting only the API thread's default cache.
- Config/docs contract: session_queue_mode description now documents
device-affinity reordering in single-user multi-GPU mode (and that
explicit FIFO disables it), and that user rotation outranks priority
across users in round_robin mode. Multi-GPU docs no longer claim
generation_devices: [] is valid, and describe shared-RAM weight
deduplication instead of per-GPU duplication.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: address multi-GPU review findings (RAM accounting, stats aggregation, MPS validation)
- SharedCpuWeightsStore.invalidate() now retires still-referenced entries
instead of dropping them from accounting, so RamBudget keeps counting
retired weights until the last locked holder releases them. Prevents
admitting models past max_cache_ram_gb while a replacement and a stale
copy are both resident.
- /models/stats aggregation takes max of cache_size and high_watermark
across per-device caches (they share one global RamBudget, so summing
over-reported an N-GPU system ~N times); event counters are still summed.
- TorchDevice.get_generation_devices() rejects 'mps' when MPS is
unavailable, so the runtime_config API 422s instead of persisting a
device that fails at first tensor op.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(frontend): lint:prettier
* fix: address JPPhoto's 2026-07-21 review (12 items)
Backend:
- layer_patcher: hold MODEL_LOAD_LOCK.read_lock() across patch application so
FLUX Control LoRA shape expansion (register_parameter) cannot overlap a
concurrent model construction's process-global init_empty_weights patch
- flux_redux/flux_denoise: store Redux conditioning on CPU (it may be produced
on a borrowed idle GPU) and assign the .to() result when consuming it
- model_cache/ram_budget: coordinate eviction across device caches — when a
cache's own stack is exhausted and the global budget is still short, peers
evict their unlocked entries (non-blocking lock, deadlock-free), so
max_cache_ram_gb holds even when RAM is retained only by an idle device
- session_queue: 'except current' operations protect the workflow-call chain of
EVERY in-progress item, not one arbitrary get_current() row
- session_queue: _cancel_in_progress_matching tolerates rows deleted by a
concurrent clear between its id SELECT and the per-item cancel
- session_processor: the post-dequeue cancel guard cancels the freshly claimed
item when skipping it (a stale cancel_event must not abandon it in_progress)
- session_processor: _clone_session_runner refuses to downgrade
DefaultSessionRunner subclasses or share custom runners across workers
- session_processor: an offloaded encoder's cache activity is attributed to the
running session's CacheStats (borrowed cache's stale stats pointer swapped
for the borrow duration)
- events: progress events report the queue item's persisted device, not the
thread-local (temporarily borrowed) one
- devices/config docs: generation_devices 'auto' defers to an explicitly
pinned legacy 'device:' setting so upgrades don't start workers on every GPU
Frontend:
- ImageViewer context: a terminal status only clears the shared progress
event/image globals when that item owns them (multi-GPU: canceling item A no
longer blanks item B's live preview)
- SettingsGenerationDevices: device tags are keyboard-operable (tabIndex +
Enter/Space activation)
Each fix has an exposure test per the review's suggestions.
* chore: regenerate openapi.json (auth on get_generation_device_options)
* fix(backend): avoid MODEL_LOAD_LOCK self-deadlock when patching a LoRA on a cold cache
apply_smart_model_patches() held MODEL_LOAD_LOCK.read_lock() across its patch
loop, but callers pass a lazy generator (e.g. flux_text_encoder._t5_lora_iterator)
that constructs each LoRA via context.models.load() on demand. A cold-cache load
takes MODEL_LOAD_LOCK.write_lock(); since the lock is non-reentrant and
write-preferring, acquiring the write lock while this same thread already holds the
read lock deadlocks (write waits for readers==0, but the consuming thread is that
reader). The generation hung silently right after the encoder/tokenizer load,
whenever a LoRA was applied and not already cached.
Materialize the patch iterable before taking the read lock so every LoRA
construction takes (and releases) the write lock first; the read lock then covers
patch application only, which is its actual purpose (FLUX Control LoRA shape
expansion calls register_parameter and must exclude concurrent construction).
Compatible with wan_denoise's per-call iterator factory, and unrelated to the SD
UNet path, which loads the LoRA before calling the singular patcher (no lock held).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address JPPhoto's four merge blockers from the 2026-07-22 review
1. model_cache: a peer whose lock is contended during cross-cache eviction
no longer leaves the shared RAM budget exceeded indefinitely.
evict_unlocked_for_peer returns None on contention; the requester records
a reconcile request on each skipped peer, and the synchronized-decorator
hook honors it as soon as the peer's current operation releases the lock
(outermost frame only — the RLock may be held reentrantly). The pending
flag stays set until the budget is actually satisfied, so overshoot held
by locked entries reconciles when their unlock releases the lock.
2. session_processor: a stale cancellation event from the previous item no
longer cancels the freshly claimed, unrelated item. The post-dequeue
guard now treats the DB status as the authority: a terminal row is
skipped; a set cancel_event with a non-terminal row is a stale signal
(a genuine cancel writes the row terminal BEFORE emitting) and is
cleared, with a post-clear terminal re-check closing the clear's own
race window. A shutdown-raced claim is still canceled so it isn't
abandoned in_progress.
3. flux2_klein_text_encoder: conditioning is detached and moved to CPU
before context.conditioning.save(), matching flux_text_encoder and
flux_redux — the node is idle_gpu_offloadable, and GPU-resident
embeddings would pin VRAM on a borrowed device after its pool lock is
released.
4. session_queue clear: user-scoped clearing no longer assumes one current
item. clear() cancels every in-progress item in scope via
_cancel_in_progress_matching (each item's own status-changed event
signals the worker running exactly that item) before deleting rows —
same pattern as delete_by_destination; the router's arbitrary
get_current() check (which could 403 the owner or cancel another user's
item) is removed; and _on_queue_cleared honors the event's user_id so a
scoped clear cannot stop other users' workers and abandon their rows.
Each fix carries the regression test JPPhoto specified: contended-peer
budget reconcile, stale-event-runs-item (plus the mid-clear race and
shutdown cases), CPU-backed Klein conditioning, and Alice/Bob concurrent
clear isolation at both the service and the event-handler layer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: regenerate OpenAPI schema for the clear endpoint docstring
The merge-blocker fix 68edb02127 reworded the clear route's docstring,
which is the OpenAPI operation description — openapi.json and schema.ts
must follow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): close lost-wakeup race in deferred RAM-budget reconcile
The deferred reconcile request was recorded pre-admission and honored only
by the peer's next lock release. Two interleavings could strand the shared
RAM budget above its cap indefinitely:
- Lost wakeup: the busy peer releases its lock (running its reconcile hook
while the flag is still unset) before request_budget_reconcile() sets the
flag; if the peer then stays idle, no future release honors the request.
- Pre-admission clearing: a peer's reconcile could run between the request
and the new model being counted, see the budget as satisfied, and clear
the flag before the admission pushed usage over the cap.
Fix both by (1) moving the reconcile request to the end of put(), after the
new model is counted, so peers always evaluate the true budget state, and
(2) having request_budget_reconcile() attempt the reconcile inline with a
non-blocking lock acquire: either the peer's lock is free now and the
reconcile runs immediately, or it is still held and the eventual release
hook — which runs strictly after the flag is set — performs it.
The prior regression test masked the race by touching cache_b.stats after
the request; it now emulates the production release hook in the holder
thread and asserts reconciliation with no subsequent cache access, and a
new test forces the lost-wakeup interleaving by delaying the request until
the peer's operation has fully finished.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): close remaining RAM-budget reconcile gaps
Addresses the three lingering issues from review of the deferred
budget-reconcile mechanism:
1. Manual lock releases bypass the reconcile hook. cached_model_keys()
and evict_unlocked_for_peer() acquire/release _lock without the
synchronized decorator, so a reconcile request whose inline attempt
failed on their held lock was stranded when they released. Both now
run the same reconcile hook after their manual release (non-blocking,
preserving cached_model_keys' no-stall guarantee and avoiding the
hold-A-block-on-B deadlock shape in evict_unlocked_for_peer).
2. clear() can wipe a concurrent request. A reconciler observing a
satisfied budget could clear the pending flag just after a peer's
admission (already counted, budget negative) set it, and the peer's
inline attempt then saw the flag unset and returned — leaving the
budget exceeded with no pending request. The reconcile now runs as a
loop with a single guarded clear site: because admissions are counted
before the flag is set, a negative budget re-check immediately after
the clear proves a request may have been wiped; the flag is restored
and reconciliation continues. This covers both former clear sites
(satisfied early-out and post-eviction).
3. No reconcile trigger when the admitting cache itself holds the
overshoot. put() requests reconciles from peers only, so when the
exceeded budget was held by the admitting cache's own locked entry,
no pending request existed anywhere and the eventual unlock ran its
hook with the flag unset. unlock() now records a reconcile request on
its own cache whenever it completes with the shared budget exceeded,
so the entry that just became evictable triggers the reconcile.
Supporting change: put() admitting a model while a peer's reconcile
request is already pending must not let its own release hook evict the
just-admitted entry before the loader's immediately-following get()
(that would break the in-flight load with an IndexError). CacheRecord
gains an awaiting_first_use grace flag, set on admission and cleared on
first get()/lock(), which the asynchronous eviction paths (budget
reconcile, peer-requested eviction) skip. The local make_room path
ignores it: cold loads are serialized under MODEL_LOAD_LOCK, so it can
never see another loader's entry inside the put()->get() window, and
this bounds the flag's lifetime if a load errors out in between.
Each new regression test was verified to fail against the previous
implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): bound the admission grace and keep cached_model_keys stall-free
Addresses the three issues from JPPhoto's 2026-07-27 review:
1. Prefetched submodels can no longer shield the budget forever. The SD
single-file loader's proactive submodel put()s are now admitted with
prefetch=True (no post-admission grace), since nothing ever get()s or
lock()s them. As a backstop, put() sweeps stale grace flags from prior
loads — cold loads are serialized under MODEL_LOAD_LOCK, so any flag
still standing at the next admission belongs to a dead load (errored
before get(), or LoadedModel dropped before lock()) and is cleared.
2. The grace now survives get() and ends at lock(). get() is synchronized,
so clearing the flag inside it let get()'s own release hook run a
pending reconcile and evict the very record it had just selected —
detaching a live model from the cache and its RAM accounting before the
caller could lock it. load_default also retrieves immediately after
put() so no failure in between can orphan a graced record.
3. cached_model_keys()'s manual-release hook hands a pending reconcile to
a short-lived background thread instead of running it inline:
reconciliation evicts models and calls gc.collect(), which would break
the method's no-stall contract and pause session dequeue.
Each new regression test verified to fail with its mechanism reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): release abandoned admission grace
* fix(model cache): keep grace release off the collecting thread
release_first_use_grace() is invoked from a weakref.finalize callback, so it
runs at an arbitrary decref/garbage-collection point in an arbitrary thread.
Making it @synchronized therefore made ModelCache._lock — and, through the
decorator's release hook, a full budget reconcile — reachable from anywhere.
That inverts the lock order RamBudget documents as impossible. The hook's
_reconcile_budget_if_pending reads RamBudget.available() ->
SharedCpuWeightsStore.total_bytes_in_use(), both plain non-reentrant locks. A
thread inside SharedCpuWeightsStore.acquire() holds the store lock while summing
tensor sizes, an allocation loop that trips generational GC; if that collection
reclaims an abandoned wrapper belonging to another device's cache, the release
hook re-enters the store lock the thread is already holding and the thread
deadlocks against itself, still holding it. Every other cache then blocks on its
next _delete_cache_entry -> release_shared_weights(). Reproduced on a two-cache
budget: the collecting thread wedges in total_bytes_in_use() and never returns.
The same hook also ran evictions, gc.collect() and empty_cache() inline in
whatever unrelated thread happened to drop the reference — including the API
event loop — undoing the no-stall contract cached_model_keys() was just given.
Do no locking work in the callback: hand the release to a short-lived background
thread, exactly as cached_model_keys() does with its own pending reconcile. The
thread may wait on the cache lock and do the slow work; the collecting thread
returns immediately.
The existing abandoned-wrapper test now polls for the (asynchronous) release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): harden the deferred grace release
Follow-ups from an adversarial review of 55a6fc496d:
- Thread.start() can raise RuntimeError under thread/process limits. A
weakref.finalize callback gets no retry (weakref retires it before invoking
it) and its exceptions go to sys.unraisablehook, so the release was silently
lost and the record kept shielding an idle cache. Fall back to clearing the
flag inline under a non-blocking acquire, which takes no store or budget lock
and so still cannot deadlock the collecting thread. No reconcile on that path
by design: a pending request stays set for the next cache operation.
- The regression test's outcome was a pure function of the ambient allocation
count: nothing pinned the cycle between its creation and the collector thread,
so an automatic gen-0 pass landing in the setup reclaimed it on the main
thread and the test passed vacuously (or tripped its own setup assertions).
Under an allocation-shifting plugin it failed at 6 of 12 offsets. Disable
automatic gc across the setup so only the explicit collect reclaims the cycle;
the same sweep is now 12 of 12 passing, and the test still fails against
7ffee4db04 with the expected re-entrancy report.
- Correct the docstring: Thread.start() waits for the child to bootstrap, so the
guarantee is "no lock waits", not "returns immediately".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): queue deferred cache work
* fix(model cache): stop the deferred worker pinning records and dying silently
Follow-ups from an adversarial review of c384ea187d, which replaced the
per-release background threads with one long-lived worker per cache.
- The worker's `work` local stays bound while it blocks in the next get(), so
the last-processed CacheRecord — and transitively its model's CPU weights —
was pinned until some unrelated item happened to be queued behind it. That is
worse than an ordinary leak: _release_first_use_grace's release hook can evict
that very record, removing it from the cache AND subtracting its bytes from
the RamBudget, so the budget under-reported a model that was still resident
and the next admission over-committed. Reproduced on a two-cache budget: after
eviction plus an explicit gc.collect(), both the record and its module were
still alive; queueing one more item freed them. The per-call threads this
replaced did not have the bug — Thread._bootstrap_inner deletes _args on exit.
Clear the reference in a finally before looping back.
- `if self._deferred_work_thread.ident is None` is a "was it ever started"
check, not a liveness check: ident is never cleared and a Thread cannot be
restarted. A worker lost to an unexpected error (a logging handler that
raises, os.fork(), or shutdown() before the first put(), which leaves its
_DEFERRED_STOP queued for the thread that put() then starts) was gone for the
life of the process, silently disabling every later grace release and budget
reconcile — the failure this mechanism exists to prevent. Create a fresh
thread whenever the previous one has exited, and never after shutdown().
- Only the worker drains the queue, but both dispatch sites enqueued
unconditionally. cached_model_keys() runs on every dequeue
(session_queue_sqlite._get_device_resident_model_keys), so an idle-device
cache that never admitted a model — and so has no worker — accumulated one
queued reconcile per dequeue forever; post-shutdown the same held for both
sites, stranding CacheRecords in a queue nothing would drain. Route both
through _dispatch_deferred, which drops the item when no worker is running.
Dropping loses nothing: such a cache has nothing to evict, and put() re-runs a
pending reconcile through the synchronized release hook when it admits one.
- shutdown()'s early return meant a keep-alive timer re-armed by a
post-shutdown put() was never cancelled by a later shutdown(). Don't arm
timers on a shut-down cache.
Tests: the two *_thread_start_failure_* tests installed their monkeypatch after
put() had already started the worker, and c384ea187d removed the only
Thread.start() from those paths — the patched raise was unreachable, so both
passed without exercising their premise. Retargeted to what they actually
verify (the finalizer and the lookup must not block, and the reconcile still
happens). Four regression tests added; each fails against c384ea187d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): stop the deferred worker outliving and pinning its cache
Second round of adversarial-review follow-ups on the deferred-work thread.
- The worker held a bound method, so a running thread — reachable from
threading._active — kept its ModelCache alive, and with it every CacheRecord
and every model's CPU weights. A cache released without shutdown() was
therefore immortal, which is the opposite of what RamBudget's weakref registry
is built for. Measured against the parent commit 7ffee4db04: five caches
dropped without shutdown() left 5/5 caches and 5/5 models resident and five
worker threads running, where the parent left 0/5 and no threads. The previous
`ident is None` guard had accidentally bounded this (a cache whose worker died
could never re-acquire a pinning thread); reviving the worker removed that
bound, so the fix has to remove the strong reference itself. The worker is now
a module-level function taking a weakref, and a weakref.finalize pushes
_DEFERRED_STOP when the cache is collected so the parked thread exits instead
of leaking one thread per abandoned cache.
- Thread.start() is called from put(), which runs under both the cache lock and
MODEL_LOAD_LOCK's write lock while completing a load. Under thread/pid
exhaustion (RLIMIT_NPROC, a container's pids.max) its RuntimeError escaped and
failed a generation whose model had already been fully constructed — to lose
an optimization that _dispatch_deferred is explicitly designed to survive the
absence of. Log and carry on; the next admission retries.
- _dispatch_deferred justified dropping work with "a cache without a worker has
never admitted a model". That was false: put() after shutdown() is reachable
in production, because Invoker.stop() stops model_manager before
session_processor, so an in-flight generation can admit a model after every
cache has been shut down — and thread exhaustion reaches the same state
without a teardown to bound it. Such a record was admitted with the first-use
grace, then permanently shielded from both asynchronous eviction paths with
its bytes still charged to the shared budget. Make the claim true instead of
rewording it: put() grants the grace only when a worker is running to release
it. lock() still clears it on the normal path, so nothing changes when the
worker is healthy.
Tests: the post-shutdown half of the drop test never set the pending flag, so
cached_model_keys() short-circuited before reaching the dispatch guard and the
assertion held unconditionally. Poll the budget in the pin test rather than
reading it the instant the key disappears (_delete_cache_entry pops before it
releases the weights). Two regression tests added; all seven of this series'
new tests fail against c384ea187d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): release abandoned RAM budget
* fix(model cache): pin LoRA patches during use
* fix(model cache): harden the LoRA pin path and close its review gaps
Review fixes for the LoRA-pinning commit (1142430f87):
- Fix the CI failure it shipped: test_krea2_text_encoder's fake
LoadedModel lacked model_in_ram(), and the encoder's LoRA iterator now
calls it. The fake now models the pin (with depth tracking), and the
test asserts the patch spec carries a working pin.
- Stop a keep-alive Timer.start() failure from leaking a permanent pin.
@record_activity runs after lock_in_ram() has incremented the lock
count but before model_in_ram()'s unlock-pairing try block is entered,
so a RuntimeError under thread/pid exhaustion would pin the record
(and its shared-budget bytes) for the life of the process. The timer
is an optimization: log and continue instead.
- Give lock_in_ram() the same already-dropped-record diagnostic as
lock()/unlock(), so a pin on a detached record produces a matching
lock-side message (issue 7513).
- Pin the LoRA cache record in LoRAExt.patch_unet (the modular-denoise
path) while its tensors are read during direct patching. This was the
one remaining producer that dropped its LoadedModel handle at load
time, leaving the record evictable by a peer cache mid-patch.
- Close test vacuities: the pin-retention test in test_layer_patcher
could not detect a dropped cache_pins.close() (the ExitStack would be
collected silently), and the cache-side pin test only covered a warm
record, leaving lock_in_ram's grace-clearing dead code under test.
Added pin-release assertions for the normal, body-raise,
restore-raise, and mid-materialization-raise paths (all verified to
fail with close() neutered), a cold-record grace/finalizer test, and
a Timer-failure regression test (verified to fail pre-fix).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): close the remaining raise-after-lock leak and widen the LoRAExt pin
Self-review fixes for d63c1f37e0:
- The synchronized-decorator's post-release reconcile hook runs inside
the caller's frame after the method body, so a raise there (e.g.
TorchDevice.empty_cache on a sick CUDA context after an eviction)
escaped lock_in_ram()/lock() after the lock count was incremented —
the same permanent-pin leak as the Timer.start() case, via a
different path. The reconcile is deferrable housekeeping: swallow and
log; the pending flag is only cleared once the budget is satisfied,
so the next lock release retries. Regression test verified to fail
pre-fix.
- LoRAExt.patch_unet's pin now spans the yielded scope, not just the
patch application: despite force_direct_patching=True, fp8-storage
modules are routed to sidecar patching (float8 weights cannot be
patched in place), which stores a live reference to the cached
patch's layers inside the UNet for the whole denoise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
Co-authored-by: JPPhoto <jpollack@jpollackphoto.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Text encoder loads can force the denoise model to unload/reload on single-device cache paths. On dual-GPU systems, keeping text encoders resident on the other CUDA device avoids that churn and makes repeated generation materially smoother.
Behavior
Verification
pnpm lint:prettierpnpm lint:tscpnpm lint:knipopenapi.jsonNotes
This branch was prepared from
upstream/mainand squashed to one focused commit. It does not include local fork/runtime update scripts, batch-specific files, or unrelated compatibility work.