feat: multi-GPU parallel session execution - #9263
Conversation
Run one generation session per configured GPU concurrently, with a tiled progress preview. Multi-user isolation is unchanged. Backed by five seams: - Per-thread device context (TorchDevice.set/get/clear_session_device); choose_torch_device() consults it first, so all device-selecting call sites resolve to the calling worker's GPU with no per-node changes. - Per-device model caches: build_model_manager builds one ModelCache per generation device; ModelLoadService.ram_cache resolves by current thread device; ram_caches fans out clear/drop/shutdown. - Atomic concurrent dequeue: a dequeue lock makes select+claim atomic so concurrent workers never claim the same item (works on FIFO; round-robin from invoke-ai#9086 slots in later). - Worker pool: one _SessionWorker per device, each pinning torch.cuda.set_device and its session device, with its own runner and cancel event; cancellation routes via an {item_id -> worker} lookup. Single-device installs keep the exact legacy single-worker behavior. Profiling disabled when >1 worker. - New config `generation_devices`; unset = legacy single-worker mode. Frontend: the canvas staging area already tiles per queue item; the main ImageViewer now tracks progress per session and renders a tile grid (ProgressImageTiles) when more than one session is active. Also adds a lock to ObjectSerializerForwardCache for concurrent access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_model_load_device_routing mutated the process-wide get_config() singleton (device = "cuda:0") to exercise the per-thread cache routing, but never restored it. The leaked CUDA device was then picked up by a later test (test_model_load::test_loading) via choose_torch_device(), which crashed with "Torch not compiled with CUDA enabled" on the CUDA-less CI runner. Add an autouse fixture to save/restore device and clear any pinned session device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n_devices Regenerate openapi.json (make frontend-openapi) and the frontend schema.ts types (make frontend-typegen) so they include the new generation_devices config field, fixing the openapi-checks and typegen-checks CI jobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`make frontend-openapi` used a bare `python` from a different environment that emitted the CacheStats @DataClass docstring as a schema description. CI generates the schema via `uv run`, which does not, so openapi-checks failed on the diff. Regenerate with the uv-locked environment to drop the stray description while keeping the generation_devices field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o prevent meta-device corruption Parallel multi-GPU session workers could intermittently crash with "unrecognized device meta" (denoise) or "Cannot copy out of meta tensor; no data!" (l2i), because model loading relies on process-global, non-thread-safe monkey-patches. accelerate.init_empty_weights() (used directly by the loaders and implicitly by diffusers' default low_cpu_mem_usage=True in from_pretrained) swaps torch.nn.Module.register_parameter globally for the duration of a load, routing every newly-registered parameter to the meta device. The model cache's VRAM load/unload runs nn.Module.load_state_dict(assign=True), whose assign path does setattr -> __setattr__ -> register_parameter. When one worker's VRAM move overlapped another worker's from_pretrained, the move's real weights got hijacked onto meta and blew up on the next .to(device). Introduce MODEL_LOAD_LOCK, a write-preferring readers-writer lock: - write lock = model construction (_load_and_cache, load_model_from_path), exclusive. - read lock = VRAM load/unload (ModelCache.lock(), repair_required_tensors_on_device). VRAM transfers across GPUs still overlap each other; they only block while a construction holds the write lock. The lock is always acquired before any per-cache lock to keep a consistent order and avoid an AB-BA deadlock with the writer's make_room/put. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ions Image.open() is lazy: it reads the header but defers pixel decoding (and holds the file handle open) until the first .load()/.copy()/.convert(). The opened object was cached and the same object handed to every caller, so in multi-GPU parallel mode two session-processor worker threads could call .copy() on it concurrently and race on the shared file handle and decoder state. This surfaced as "broken data stream when reading image file" and "AssertionError: self.png is not None" during inpainting with batch >1. Force the decode (image.load()) before the object enters the cache so the cached object is safe for concurrent reads, and guard the cache structures (__cache / __cache_ids) with a lock since they are now mutated from multiple threads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generation progress bars (under the Invoke button and the Viewer tab) both read a single global $lastProgressEvent atom, which every session overwrites. With parallel multi-GPU sessions this made the bar jump back and forth between sessions. Track progress per queue item id and render one bar per in-flight session, stacked vertically, each removed as its session reaches a terminal state. - stores.ts: add $progressEvents (map keyed by item_id), $activeProgressEvents (sorted), and set/clear helpers. - setEventListeners.tsx: populate per-item progress on invocation_progress; clear per item on terminal status; clear all on connect/disconnect/queue cleared. - ProgressBar.tsx: render a vertical stack of bars (one per active session) with a single-bar fallback for the idle / model-loading window; add containerProps so dockview tabs can position the stack. - Dockview tab call sites: move positioning into containerProps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
$progressEvents is only referenced within stores.ts (via the $activeProgressEvents computed and the set/clear helpers), so exporting it tripped knip's unused-exports check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With 4 GPUs the stacked per-session progress bars grew past the bottom strip of the dockview tab and overlapped the "Viewer" label. Add a fitHeightPx prop: in fit mode the stack is capped to the available strip (10px below the ~40px tab's centered label) and the bars flex to share it, shrinking below their natural height only once they no longer fit. With 1-2 sessions the bars keep their familiar thin height; with 3+ they scale down to stay within the strip. The sidebar bar is unaffected and continues to stack at natural height (it has the vertical room). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fault generation_devices now accepts "auto" (the new default), which expands to every visible CUDA device — so multi-GPU parallel generation works out of the box without manually listing devices. On GPU-less systems "auto" resolves to the single cpu/mps device, preserving serial behavior. - config_default.py: type is now Union[Literal["auto"], list[str]], default "auto"; validator accepts "auto" or a list of device strings. - devices.py: add TorchDevice.get_generation_devices(), the single resolver that expands "auto", normalizes, and deduplicates. - session_processor / model_manager: both consumers use the resolver instead of iterating the raw config value (which would have iterated the characters of the "auto" string). - Regenerated docs/src/generated/settings.json. - Tests for the resolver (auto-with/without-CUDA, dedup, empty). An explicit single-device list (e.g. [cuda:0]) or an empty list opts out of parallelism. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
added
api
python
|
An explanation of the latest commit: Previously, Now each
Per-GPU execution remains parallel. Each GPU session worker and each per-device |
…silently Follow-ups from an adversarial review of c384ea1, which replaced the per-release background threads with one long-lived worker per cache. - The worker's `work` local stays bound while it blocks in the next get(), so the last-processed CacheRecord — and transitively its model's CPU weights — was pinned until some unrelated item happened to be queued behind it. That is worse than an ordinary leak: _release_first_use_grace's release hook can evict that very record, removing it from the cache AND subtracting its bytes from the RamBudget, so the budget under-reported a model that was still resident and the next admission over-committed. Reproduced on a two-cache budget: after eviction plus an explicit gc.collect(), both the record and its module were still alive; queueing one more item freed them. The per-call threads this replaced did not have the bug — Thread._bootstrap_inner deletes _args on exit. Clear the reference in a finally before looping back. - `if self._deferred_work_thread.ident is None` is a "was it ever started" check, not a liveness check: ident is never cleared and a Thread cannot be restarted. A worker lost to an unexpected error (a logging handler that raises, os.fork(), or shutdown() before the first put(), which leaves its _DEFERRED_STOP queued for the thread that put() then starts) was gone for the life of the process, silently disabling every later grace release and budget reconcile — the failure this mechanism exists to prevent. Create a fresh thread whenever the previous one has exited, and never after shutdown(). - Only the worker drains the queue, but both dispatch sites enqueued unconditionally. cached_model_keys() runs on every dequeue (session_queue_sqlite._get_device_resident_model_keys), so an idle-device cache that never admitted a model — and so has no worker — accumulated one queued reconcile per dequeue forever; post-shutdown the same held for both sites, stranding CacheRecords in a queue nothing would drain. Route both through _dispatch_deferred, which drops the item when no worker is running. Dropping loses nothing: such a cache has nothing to evict, and put() re-runs a pending reconcile through the synchronized release hook when it admits one. - shutdown()'s early return meant a keep-alive timer re-armed by a post-shutdown put() was never cancelled by a later shutdown(). Don't arm timers on a shut-down cache. Tests: the two *_thread_start_failure_* tests installed their monkeypatch after put() had already started the worker, and c384ea1 removed the only Thread.start() from those paths — the patched raise was unreachable, so both passed without exercising their premise. Retargeted to what they actually verify (the finalizer and the lookup must not block, and the reconcile still happens). Four regression tests added; each fails against c384ea1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cache Second round of adversarial-review follow-ups on the deferred-work thread. - The worker held a bound method, so a running thread — reachable from threading._active — kept its ModelCache alive, and with it every CacheRecord and every model's CPU weights. A cache released without shutdown() was therefore immortal, which is the opposite of what RamBudget's weakref registry is built for. Measured against the parent commit 7ffee4d: five caches dropped without shutdown() left 5/5 caches and 5/5 models resident and five worker threads running, where the parent left 0/5 and no threads. The previous `ident is None` guard had accidentally bounded this (a cache whose worker died could never re-acquire a pinning thread); reviving the worker removed that bound, so the fix has to remove the strong reference itself. The worker is now a module-level function taking a weakref, and a weakref.finalize pushes _DEFERRED_STOP when the cache is collected so the parked thread exits instead of leaking one thread per abandoned cache. - Thread.start() is called from put(), which runs under both the cache lock and MODEL_LOAD_LOCK's write lock while completing a load. Under thread/pid exhaustion (RLIMIT_NPROC, a container's pids.max) its RuntimeError escaped and failed a generation whose model had already been fully constructed — to lose an optimization that _dispatch_deferred is explicitly designed to survive the absence of. Log and carry on; the next admission retries. - _dispatch_deferred justified dropping work with "a cache without a worker has never admitted a model". That was false: put() after shutdown() is reachable in production, because Invoker.stop() stops model_manager before session_processor, so an in-flight generation can admit a model after every cache has been shut down — and thread exhaustion reaches the same state without a teardown to bound it. Such a record was admitted with the first-use grace, then permanently shielded from both asynchronous eviction paths with its bytes still charged to the shared budget. Make the claim true instead of rewording it: put() grants the grace only when a worker is running to release it. lock() still clears it on the normal path, so nothing changes when the worker is healthy. Tests: the post-shutdown half of the drop test never set the pending flag, so cached_model_keys() short-circuited before reaching the dispatch guard and the assertion held unconditionally. Poll the budget in the pin test rather than reading it the instant the key disappears (_delete_cache_entry pops before it releases the weights). Two regression tests added; all seven of this series' new tests fail against c384ea1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Merged Merge with
|
| caches alive | models alive | worker threads | |
|---|---|---|---|
7ffee4db04 |
0/5 | 0/5 | 0 |
c384ea18 |
5/5 | 5/5 | 5 |
Production shuts every cache down in ModelManagerService.stop(), so the blast radius there is non-clean exits — but it also leaked threads and models across the test suite, and it defeated the weakref registry RamBudget is built on. The worker is now a module-level function taking a weakref.ref, with a weakref.finalize that pushes _DEFERRED_STOP so the parked thread exits instead of leaking one per abandoned cache.
3. ident is None was a "was it ever started" check, not a liveness check
Thread.ident is never cleared and a Thread can't be restarted, so a worker lost to an unexpected error was gone for the life of the process — silently disabling every later grace release and budget reconcile, i.e. exactly the "release is lost and the record keeps shielding an idle cache" failure c7dddb6 was written to prevent. Reachable with no injected fault: shutdown() before the first put() leaves its _DEFERRED_STOP queued for the thread that put() then starts, which consumes it and exits.
4. Both dispatch sites enqueued into a queue only the worker drains
cached_model_keys() runs on every dequeue (session_queue_sqlite._get_device_resident_model_keys), so an idle-device cache that never admitted a model — and therefore has no worker — accumulated one queued reconcile per dequeue, forever. Post-shutdown the same held for both sites, stranding CacheRecords in a queue nothing would drain.
5. A Thread.start() failure failed the generation
put() runs under both the cache lock and MODEL_LOAD_LOCK's write lock while completing a load, so a RuntimeError: can't start new thread (RLIMIT_NPROC, a container's pids.max) aborted a load whose model had already been fully constructed — to lose an optimization the dispatch path is explicitly designed to survive the absence of. Now logged and skipped; the next admission retries.
6. The justification for dropping deferred work was false
_dispatch_deferred reasoned "a cache without a worker has never admitted a model, so it has nothing to release or evict". Not true: Invoker.stop() iterates vars(self.services), and model_manager is stopped before session_processor, so an in-flight generation can put() after every cache has been shut down. Thread exhaustion reaches the same state with no teardown to bound it. Such a record was admitted with awaiting_first_use=True and then permanently invisible to both async eviction paths while its bytes stayed charged to the shared budget.
Rather than reword the comment, I made the claim true: put() grants the first-use grace only when a worker is alive to release it. lock() still clears it on the normal path, so nothing changes when the worker is healthy — and a dropped CacheRecord is now one that was never shielded in the first place.
7. shutdown()'s early return orphaned a keep-alive timer
A post-shutdown put() re-armed the timer, and the idempotent shutdown() never cancelled it. Fixed at the source — a shut-down cache doesn't arm timers.
Tests
The two *_thread_start_failure_* tests were vacuous: both installed their Thread.start monkeypatch after put() had already started the worker, and c384ea1 had removed the only Thread.start() from those paths, so the injected failure was unreachable. They'd have passed with or without any start-failure handling. Retargeted to what they actually verify (the finalizer and the lookup must not block, and the reconcile still happens).
Seven regression tests added; each one fails against c384ea1. Cache suite run 10× with no flakes; full backend suite green at 3729 passed, 118 skipped.
Known limitations, deliberately not fixed
- Dropping a
ModelCacheleaves aSharedCpuWeightsStorerefcount behind. I checked — identical on7ffee4db04, so it's pre-existing and orthogonal to this series. - A
shutdown()landing between_dispatch_deferred's liveness check and itsput()can strand a single item. Closing it needs a lock that method can't take (it runs inside aweakref.finalizecallback), and the cost is one record during teardown. Documented in the code.
# Conflicts: # invokeai/app/services/session_processor/session_processor_default.py # invokeai/app/services/session_queue/session_queue_sqlite.py
|
Resolved as #9355's single-transaction structure carrying this branch's Re-verified after the second merge: backend 3761 passed, frontend 1675 passed, |
|
Fixing now:
|
|
@lstein I am going to work on this merge blocker:
Please add the rest of these in a follow-up PR:
|
Review fixes for the LoRA-pinning commit (1142430): - Fix the CI failure it shipped: test_krea2_text_encoder's fake LoadedModel lacked model_in_ram(), and the encoder's LoRA iterator now calls it. The fake now models the pin (with depth tracking), and the test asserts the patch spec carries a working pin. - Stop a keep-alive Timer.start() failure from leaking a permanent pin. @record_activity runs after lock_in_ram() has incremented the lock count but before model_in_ram()'s unlock-pairing try block is entered, so a RuntimeError under thread/pid exhaustion would pin the record (and its shared-budget bytes) for the life of the process. The timer is an optimization: log and continue instead. - Give lock_in_ram() the same already-dropped-record diagnostic as lock()/unlock(), so a pin on a detached record produces a matching lock-side message (issue 7513). - Pin the LoRA cache record in LoRAExt.patch_unet (the modular-denoise path) while its tensors are read during direct patching. This was the one remaining producer that dropped its LoadedModel handle at load time, leaving the record evictable by a peer cache mid-patch. - Close test vacuities: the pin-retention test in test_layer_patcher could not detect a dropped cache_pins.close() (the ExitStack would be collected silently), and the cache-side pin test only covered a warm record, leaving lock_in_ram's grace-clearing dead code under test. Added pin-release assertions for the normal, body-raise, restore-raise, and mid-materialization-raise paths (all verified to fail with close() neutered), a cold-record grace/finalizer test, and a Timer-failure regression test (verified to fail pre-fix). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the LoRAExt pin Self-review fixes for d63c1f3: - The synchronized-decorator's post-release reconcile hook runs inside the caller's frame after the method body, so a raise there (e.g. TorchDevice.empty_cache on a sick CUDA context after an eviction) escaped lock_in_ram()/lock() after the lock count was incremented — the same permanent-pin leak as the Timer.start() case, via a different path. The reconcile is deferrable housekeeping: swallow and log; the pending flag is only cleared once the budget is satisfied, so the next lock release retries. Regression test verified to fail pre-fix. - LoRAExt.patch_unet's pin now spans the yielded scope, not just the patch application: despite force_direct_patching=True, fp8-storage modules are routed to sidecar patching (float8 weights cannot be patched in place), which stores a live reference to the cached patch's layers inside the UNet for the whole denoise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Adversarial review of 1142430 ("fix(model cache): pin LoRA patches during use"). Three independent fresh-context reviews (state lifecycle; locking/deadlock/concurrency; callers, coverage and test quality), plus a fourth adversarial pass over the fixes themselves. Every finding verified against the code before acting. Head is now First: the mechanism is right. The 3-tuple Confirmed defects, fixed in d63c1f31. The commit shipped a red CI
All six python-tests matrix jobs failed at this SHA. The fake now models the pin with depth tracking, and the test asserts the yielded spec carries a working pin — which also makes it the first invocation-level test to defend the 3-tuple invariant at all. 2. A keep-alive
|
…ancel Two concurrent bulk cancellations could select the same in-progress row; the first canceled it and the second, seeing the already-terminal row returned by _set_queue_item_status(), counted it again — both responses reporting that they canceled the same item. The terminal guard and the UPDATE already ran atomically (a single transaction under SqliteDatabase's process-wide lock); what was missing was exposing WHO performed the transition. _set_queue_item_status() now delegates to _transition_queue_item_status(), which additionally returns whether THIS call moved the row, and _cancel_in_progress_matching() counts only actual transitions — so exactly one racing caller observes itself as the canceller. The docstring records that the atomicity comes from the database lock, not the SQL, so a future move to per-thread connections doesn't silently reintroduce the double count. Includes a two-thread barrier test (in-memory SQLite) asserting each item is counted exactly once across concurrent bulk cancels; the pre-fix counter reports 4 for 2 items under the same interleaving. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… auto copy The schema description, Settings UI copy, and configuration guide's behavior table all said `generation_devices: auto` uses every available GPU, but TorchDevice.get_generation_devices() deliberately resolves it to the single pinned legacy `device` when one is set — only a later docs note disclosed the exception. An upgraded install with `device: cuda:1` displayed "Auto (all GPUs)" while starting one worker. - config_default.py: the field description now states the precedence (and that an explicit list overrides `device`). - Settings UI: the badge is "Auto" (not "Auto (all GPUs)") and the help text explains the legacy-device exception and the override. - invokeai-yaml.mdx: the behavior table row for `auto` states the exception where the value is introduced, not only in the notes. - Regenerated docs/src/generated/settings.json, openapi.json, schema.ts from the description source. - New test asserts all four copy locations describe the precedence. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… queue lifecycle events Two related lifecycle gaps in the image viewer's multi-session preview state (ImageViewer/context.tsx): 1. Terminal-owner fallback: when the session owning the shared $progressEvent/$progressImage globals reached a terminal state, its tile was removed but the globals were cleared (or parked on the finished session's stale frame via the resolve illusion). Since the tiled view only renders with >1 active session, the remaining session's still-running preview disappeared until its next image event. The globals are now handed to the most recently updated remaining session immediately, for every terminal status. 2. Stale lifecycle: $progressData was cleaned only by per-item terminal events. It is now also cleared on queue_cleared (scoped like workflowExecutionCoordinator.onQueueCleared, and marking the cleared items finished so a trailing progress event from a worker stopped only by the clear cannot repopulate the preview), on socket disconnect, and on $socket replacement (auth-token/user change). The store logic is extracted into viewerProgressLifecycle.ts so it can be unit tested without rendering; the provider keeps the socket subscriptions and ownership/scope checks. 16 new vitest cases cover promotion across terminal statuses and auto-switch modes, non-owner termination, clear scoping (own/unscoped/foreign/sanitized), and disconnect resets. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@JPPhoto all five non-blocking items now live in stacked draft PRs (each based on this branch's current head
All four were rebased onto / built against Also closed #9397 — your 1142430 supersedes it with a strictly stronger mechanism (a real cache pin instead of just keeping the wrapper alive). |
JPPhoto
left a comment
JPPhoto
left a comment
There was a problem hiding this comment.
Approved, and I'm assuming you've done adequate single- and multi-GPU testing.
… queue lifecycle events Two related lifecycle gaps in the image viewer's multi-session preview state (ImageViewer/context.tsx): 1. Terminal-owner fallback: when the session owning the shared $progressEvent/$progressImage globals reached a terminal state, its tile was removed but the globals were cleared (or parked on the finished session's stale frame via the resolve illusion). Since the tiled view only renders with >1 active session, the remaining session's still-running preview disappeared until its next image event. The globals are now handed to the most recently updated remaining session immediately, for every terminal status. 2. Stale lifecycle: $progressData was cleaned only by per-item terminal events. It is now also cleared on queue_cleared (scoped like workflowExecutionCoordinator.onQueueCleared, and marking the cleared items finished so a trailing progress event from a worker stopped only by the clear cannot repopulate the preview), on socket disconnect, and on $socket replacement (auth-token/user change). The store logic is extracted into viewerProgressLifecycle.ts so it can be unit tested without rendering; the provider keeps the socket subscriptions and ownership/scope checks. 16 new vitest cases cover promotion across terminal statuses and auto-switch modes, non-owner termination, clear scoping (own/unscoped/foreign/sanitized), and disconnect resets. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ancel Two concurrent bulk cancellations could select the same in-progress row; the first canceled it and the second, seeing the already-terminal row returned by _set_queue_item_status(), counted it again — both responses reporting that they canceled the same item. The terminal guard and the UPDATE already ran atomically (a single transaction under SqliteDatabase's process-wide lock); what was missing was exposing WHO performed the transition. _set_queue_item_status() now delegates to _transition_queue_item_status(), which additionally returns whether THIS call moved the row, and _cancel_in_progress_matching() counts only actual transitions — so exactly one racing caller observes itself as the canceller. The docstring records that the atomicity comes from the database lock, not the SQL, so a future move to per-thread connections doesn't silently reintroduce the double count. Includes a two-thread barrier test (in-memory SQLite) asserting each item is counted exactly once across concurrent bulk cancels; the pre-fix counter reports 4 for 2 items under the same interleaving. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… auto copy The schema description, Settings UI copy, and configuration guide's behavior table all said `generation_devices: auto` uses every available GPU, but TorchDevice.get_generation_devices() deliberately resolves it to the single pinned legacy `device` when one is set — only a later docs note disclosed the exception. An upgraded install with `device: cuda:1` displayed "Auto (all GPUs)" while starting one worker. - config_default.py: the field description now states the precedence (and that an explicit list overrides `device`). - Settings UI: the badge is "Auto" (not "Auto (all GPUs)") and the help text explains the legacy-device exception and the override. - invokeai-yaml.mdx: the behavior table row for `auto` states the exception where the value is introduced, not only in the notes. - Regenerated docs/src/generated/settings.json, openapi.json, schema.ts from the description source. - New test asserts all four copy locations describe the precedence. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… queue lifecycle events Two related lifecycle gaps in the image viewer's multi-session preview state (ImageViewer/context.tsx): 1. Terminal-owner fallback: when the session owning the shared $progressEvent/$progressImage globals reached a terminal state, its tile was removed but the globals were cleared (or parked on the finished session's stale frame via the resolve illusion). Since the tiled view only renders with >1 active session, the remaining session's still-running preview disappeared until its next image event. The globals are now handed to the most recently updated remaining session immediately, for every terminal status. 2. Stale lifecycle: $progressData was cleaned only by per-item terminal events. It is now also cleared on queue_cleared (scoped like workflowExecutionCoordinator.onQueueCleared, and marking the cleared items finished so a trailing progress event from a worker stopped only by the clear cannot repopulate the preview), on socket disconnect, and on $socket replacement (auth-token/user change). The store logic is extracted into viewerProgressLifecycle.ts so it can be unit tested without rendering; the provider keeps the socket subscriptions and ownership/scope checks. 16 new vitest cases cover promotion across terminal statuses and auto-switch modes, non-owner termination, clear scoping (own/unscoped/foreign/sanitized), and disconnect resets. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: don't hold VRAM in an idle server (CUDA context at startup) Since #9263, a freshly started server held ~128-256 MiB of VRAM per GPU before serving any request. Two startup paths were responsible: - ModelCache.__init__ sized the RAM cache via torch.cuda.mem_get_info(), which creates a CUDA context on the execution device — and #9263 now builds one ModelCache per generation device at startup. Total VRAM is read from cudaGetDeviceProperties instead, which reports the same value (verified byte-identical) without creating a context. - Each session-processor worker called torch.cuda.set_device() at thread start (i.e. at boot). The CUDA-side pin is now deferred until the worker claims its first queue item; the pin is per-thread and the thread persists, so pinning before the first item is equivalent. The TorchDevice.set_session_device() threadlocal pin, which drives all device-selection logic, still happens at thread start. Verified on a live server: with this change the process no longer appears in nvidia-smi compute-apps while idle, and queue items still process correctly on the lazily-pinned worker. Closes #9413 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cover idle CUDA context prevention --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com>
3 participants
Summary
This PR adds multi-GPU parallel generation: on a machine with more than one GPU, InvokeAI runs several generation sessions concurrently — one per GPU — instead of draining the queue one job at a time. Jobs are distributed fairly across users so a single user's large batch can't monopolize every GPU while others wait.
It's controlled by a new
generation_devicesconfig setting (defaults toauto= use every available CUDA GPU). Setting it to a single device, or leaving CUDA out of the picture, preserves the previous serial behavior exactly. The choice of GPUs can also be controlled via a new section of the Settings dialogue (restart required to take effect).Demo (turn on the sound!)
invoke-mgpu.mp4
How it works — the change is built around five small backend seams plus a frontend update, rather than per-node edits:
invokeai/backend/util/devices.py): a thread-localset/get/clear_session_deviceonTorchDevice;choose_torch_device()consults it first. This is the lynchpin — the ~79 existing call sites resolve to the worker's GPU with no per-node changes.model_manager_default,model_load_default): oneModelCacheper device, resolved by the current thread's device, with fan-out for clear/drop/shutdown. Model construction is serialized against VRAM moves to prevent meta-device corruption. A single global RAM budget is shared across the per-device caches, and identical CPU weights are deduplicated across devices (see RAM management below).session_queue_sqlite.dequeue): a lock makes select+claim atomic so concurrent workers never grab the same queue item.session_processor_default): one_SessionWorkerper device, each pinningtorch.cuda.set_device+ the session device, with its own runner and cancel event; cancellation is routed per item. Profiling is disabled when more than one worker is active.LocktoObjectSerializerForwardCacheand madeDiskImageFileStoragethread-safe for parallel sessions.Frontend: during parallel generation the progress display stacks one progress bar per active session (each disappears as its session finishes), and the image viewer tiles per-session progress previews when ≥2 sessions are active.
Idle-GPU text-encoder offload
When more than one generation device is configured and a GPU is idle, a session's text/prompt encoder runs on the idle GPU instead of the one running its denoise pipeline. This avoids evicting the denoise model from VRAM to make room for the encoder, and lets a cached encoder be reused across generations. Under full load (no idle GPU) behavior is unchanged. Controlled by
offload_text_encoders_to_idle_gpus(default on); inspired by #9310.backend/util/device_pool.py):GENERATION_DEVICE_POOLgives each generation device one exclusive-use lock. A native session blocking-acquires its own device's lock for the whole run; an encoder node try-borrows an idle device's lock for the duration of that node. A borrowed encoder and a native session are therefore mutually exclusive on a GPU — preventing the shared-encoder corruption that produced garbled images — and the design is deadlock-free (borrows are non-blocking; a session only ever blocks on its own device).@invocation(idle_gpu_offloadable=True), mirroring the existingbottleneckClassVar. Applied to the text/prompt-encoder nodes (compel + sdxl/refiner, flux, sd3, qwen-image, anima, cogview4, flux2 klein, z-image, flux_redux). The runner re-pins the worker thread to the borrowed device for the node; conditioning is stored on the CPU so the denoiser picks it up on its own GPU afterward.RAM management for parallel sessions
Running N sessions in parallel multiplies memory pressure, so this PR also makes the model cache parallel-aware:
load_state_dict(assign=True)) instead of re-reading from disk and materializing a second copy. This is loader-agnostic and now also covers GGUF models —GGMLTensordoesn't implementaten.empty_like, which previously made the largest quantized models (e.g. a Q8_0 transformer) silently re-load on every device and spike RAM; the adoptedGGMLTensorshares the quantized storage, so it's one copy across devices.Generation Devices settings refinements
A few small fixes to the Generation Devices selector and its logging:
#Nsuffix on identically-named GPUs is now tied to each device's cuda index (its position in the full available-device set) rather than its position in the possibly-filteredgeneration_deviceslist. Previously, disabling e.g.cuda:1renumbered the survivors in the backend startup log (cuda:2became#2), disagreeing with the frontend, which always labels over the full set. Now both stay consistent —cuda:2remains#3.generation_devicesonly takes effect after a restart.Related Issues / Discussions
QA Instructions
On a multi-GPU machine:
generation_devices: auto), enqueue a batch larger than the GPU count and confirm multiple sessions run simultaneously (one per GPU), with stacked progress bars and tiled previews in the viewer.generation_devices: [cuda:0]and confirm generation runs serially, exactly as before this PR.generation_devices: [cuda:0, cuda:2]and confirm only those devices are used.autoresolves to the one best device and behavior is unchanged.Running ... on idle device ...), and that the denoise model is not evicted to load the encoder. Setoffload_text_encoders_to_idle_gpus: falseand confirm the encoder runs on the session's own GPU.Adopted shared CPU weights ...log on the second device rather than a second disk load.New automated tests cover device routing (
test_model_load_device_routing.py), dequeue concurrency (test_session_queue_dequeue_concurrency.py), device resolution (test_devices.py), the device-pool lock semantics and offload mutual-exclusion (test_device_pool.py,test_encoder_offload.py), and cross-device weight adoption incl. GGUF (test_shared_weight_adoption.py).Merge Plan
Standard merge. No DB schema or redux migrations. Touches the session processor and model cache, so worth a careful look from those areas' owners.
The idle-GPU text-encoder offload (originally prototyped as a follow-on PR) is now included in this branch, along with the cross-device GGUF weight de-duplication that keeps parallel-session RAM bounded.
Checklist
What's Newcopy (if doing a release after this PR)