Repository navigation
QwenImage21Pipeline: image editing ignores the instruction and returns a haloed near-copy at output_resolution=1024 (default); 512/768 edit correctly #14824
Description
Describe the bug
With QwenImage21Pipeline, image editing at the default output_resolution=1024 ignores the instruction and returns an over-sharpened, haloed, over-saturated near-copy of the input image. The same call with output_resolution=512 or 768 produces a correct, clean edit.
This does not depend on the prefix KV cache (use_kv_cache=False gives the same image), on guidance (true_cfg_scale=4.0 adds a red band where the scarf should be but keeps the halo), on the transformers version (5.14.1 and 5.17.0 produce bit-identical results), or on precision (fp32 end to end gives the same image as bf16 to 43 dB PSNR).
An independent re-implementation of this pipeline in another framework (Swift/MLX, block-level parity with the diffusers transformer at fp32) reproduces the 1024² output from the same noise to 36 dB (bf16) / 54 dB (fp32) PSNR, so the behaviour follows from the pipeline's math as written rather than from a backend. I have only tested on Apple silicon (MPS), so a CUDA confirmation would be useful.
Observed at output_resolution=1024 for two prompts ("Make the dog wear a red scarf", "Change the background to a sunset beach") on a natural photo generated by the pipeline itself, and for a synthetic image; at 320² and 512²/768² the same edits are correct.
Questions this raises, in case they help locate it:
- Is 1024² (4,096 condition + 4,096 target tokens) inside the trained editing regime, or should
output_resolutiondefault lower / be capped for edits? QwenImage21Ropeadvances the frame position bymax(h, w)after each image block, so the condition→target frame offset grows with the grid (20 at 320², 64 at 1024²). Is that the training convention?calculate_shiftis fedlatents.shape[1](target tokens only) while the joint sequence is twice that for a 1-image edit. Intended?
Reproduction
import torch
from diffusers import QwenImage21Pipeline
pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", torch_dtype=torch.bfloat16).to("mps") # or "cuda"
# 1) a natural photo from the model itself
photo = pipe(
prompt="A photo of a golden retriever sitting on a wooden dock by a calm lake, soft morning daylight, shallow depth of field",
width=1024, height=1024, num_inference_steps=40,
generator=torch.Generator("cpu").manual_seed(42),
).images[0]
photo.save("photo.png")
# 2) the same edit at three output resolutions
for res in (512, 768, 1024):
img = pipe(
prompt="Make the dog wear a red scarf",
image=photo, output_resolution=res, num_inference_steps=40,
generator=torch.Generator("cpu").manual_seed(42),
).images[0]
img.save(f"scarf_{res}.png")
# variants that do NOT change the 1024 result:
# use_kv_cache=False
# true_cfg_scale=4.0, negative_prompt=" " (partial: a red band appears, halo remains)
# torch_dtype=torch.float32 end to endResults (40 steps, seed 42, same inputs):
| output_resolution | use_kv_cache | true_cfg_scale | dtype | result |
|---|---|---|---|---|
| 512 | True | 1.0 | bf16 | correct: red scarf, photo otherwise intact |
| 768 | True | 1.0 | bf16 | correct |
| 1024 (default) | True | 1.0 | bf16 | instruction ignored; haloed, over-sharpened near-copy of the input |
| 1024 | False | 1.0 | bf16 | identical to the row above |
| 1024 | True | 4.0 | bf16 | scarf-like red band, halo remains |
| 1024 | True | 1.0 | fp32 | same halo (43.4 dB PSNR vs the bf16 render) |
Text-to-image at 1024² and 2048² is fine with the same install; the text encoder output is unchanged between transformers 5.14.1 and 5.17.0 (bit-identical hidden states for text-only and image-conditioned prompts).
Logs
No errors or warnings are emitted; the pipeline completes normally in every configuration.
System Info
- diffusers main @ 80c7ed2 (includes Add Qwen-Image 2.1 #14804)
- transformers 5.14.1 and 5.17.0 (both tested, identical results)
- torch 2.11.0 / 2.14.0, device mps (Apple M5 Max, 128 GB), Python 3.12
- Model: Qwen/Qwen-Image-2.1 (main)
Who can help?
@yiyixuxu @sayakpaul (authors of #14804)
Activity
sayakpaul commentedon Sep 21, 2026
@naykun
peterc commentedon Sep 21, 2026
I can confirm on CUDA (RTX 3090 Ti) running diffusers @80c7ed2, torch 2.14.0+cu130, transformers 5.17.0, bf16 - running off of Python code driving it directly, no ComfyUI or anything like that. Slight difference though, no "haloed" copies but it replaces the subject entirely, e.g. "change the background" loses the dog or sometimes turns it into a cat. Seed makes no difference here - tried several.
It feels like the same or at least an adjacent problem as below 768 is also fine here. That said, I honed in a bit further and found the actual breaking point on mine is 896x896 (864x864 is fine) so from 52x52 latent to 54x54 latent.
sayakpaul commentedon Sep 21, 2026
Just to confirm, do other platforms have this issue? The pipeline was contributed by the Qwen team. So, we will wait for them to have a look.
Out of curiosity, does changing prompts, using actual CFG (
true_cfg_scale > 1), etc. -- does any of that help or not really?xocialize commentedon Sep 21, 2026
Answering both questions, plus a cross-check with an independent implementation.
Other platforms / implementations. Yes. @peterc confirms on CUDA above. On my side the decisive test was ComfyUI's native Qwen-Image 2.1 path (master @
c194dd00, no diffusers code — its own block-causal segments, prefix cache and RoPE layout): it reproduces the same 1024² degradation on the same photo and prompt, 40.1 dB PSNR to the diffusers output using ComfyUI's own seed-42 noise and its fixed shift-0.69 schedule, and 40.8 dB when I inject the diffusers noise and the exact sigmas. Same for "Change the background to a sunset beach" (39.7 dB). A third implementation (a Swift/MLX port of this pipeline) agrees with diffusers to 53.6 dB at fp32. So: three implementations, two backends, same behaviour.Worth noting that at 1024² the result is essentially noise-independent: two ComfyUI runs with different noise agree at 49.1 dB, versus 33.2 dB between two draws at 768² where the edit is correct. The output is dominated by the reference image rather than by the sampling trajectory.
Prompts / CFG. Neither helps. Two different instructions ("Make the dog wear a red scarf", "Change the background to a sunset beach") fail the same way at 1024².
true_cfg_scale=4.0with a blank negative prompt is partial: a red band appears where the scarf should be, but the halo and over-sharpening remain. Also ruled out:use_kv_cache=False, fp32 end to end, and transformers 5.14.1 vs 5.17.0 (bit-identical hidden states).Boundary. On my fixture, 896² and 960² targets edit correctly and 1024² does not — at the 16× VAE that is 56 and 60 latents per side working, 64 failing. @peterc sees the break lower (896² broken, 864² fine, i.e. 56 vs 54 latents). So the threshold looks fixture- and platform-dependent rather than a hard cutoff, but both land in the same regime, and his symptom is a more drastic version of the same failure (subject replaced rather than preserved-but-haloed).
One probe that may help localize it: a 768² reference paired with a 1024² target still degrades — 6,400 joint tokens there versus 8,192 at 1024/1024. So the trigger tracks the target grid extent, not the joint sequence length and not the reference grid. Text-to-image at 1024² and 2048² is fine, so it is specific to the reference-conditioned regime at that target extent.
That is what makes me think the pipeline is faithfully reproducing the published recipe and the open question is for the model authors: is a 64×64-latent edit target inside the trained regime? If not, capping
output_resolutionfor edits (960 is clean here) would be a reasonable interim default.peterc commentedon Sep 21, 2026
As with @xocialize above, a CFG of 4.0 had no impact on the overall problem for me. I should note that I can't confirm if my issue is exactly the same as @xocialize's but it's very similar/'adjacent' :)
I, too, did some noodling and found that it fails if, and only if, the condition image's latent grid has max(h, w) >= 56, token count. Target size (target 1024 is fine, say) and cfg/seed/steps/kv cache have no impact. (On reflection, I note this differs from what @xocialize says above.)
I ran Codex over all of this (so this is where I fall off the rails in understanding.. I am merely parroting now ;-)) and its theory is it's something to do with the "vision tower"'s training on a 48x48 patch grid (so 768x768). Make of that what you will.
Update: I switched to a simpler "blue cube"/"add a red sphere" type example and it WORKS at 1024, BTW. So that confuses things somewhat. I'm still digging, but I'll stop reporting new stuff here at this point, as I just remembered this isn't a Qwen Image 2.1 bug tracker and so this may not even be a diffusers-related problem 😂
naykun commentedon Sep 21, 2026
Hi @peterc @xocialize, I think the main issue is that the same seed is being used for both the initial image generation and the subsequent editing. Try using different seeds for the text-to-image (t2i) and image-to-image (i2i) steps, and let's see if that resolves the problem.
naykun commentedon Sep 21, 2026
The issue is predominantly observed at the 1024 resolution because you set the resolution to 1024x1024 during the t2i step. If the output of the edit step is also exactly 1024, it means the initial noise is completely identical, and the dynamic scheduler will compute the same sigma values. This exacerbates the problem we refer to as "noise replay".
peterc commentedon Sep 21, 2026
I'd be interested to see if that fixes @xocialize's problem. In my case, the t2i was at 1312 and the edit seeds were different to the one used for that (as the t2i script I use randomizes seeds). I did just re-run and verify with totally different figures, though.
However, I am going to move on as I have run numerous other image editing runs successfully at 1024 since. The only ones that have failed so far are a fox/"put a red hat on" example and the dog one @xocialize provided above.. but doing it with people, objects, landscapes, all fine. I am rarely editing photos of canids, so it may be a very isolated problem.. 😅
xocialize commentedon Sep 22, 2026
@naykun is right, and one thing in my earlier comment was wrong. Thank you — this fully explains my fixture.
Confirmed. The photo I was editing was generated by the pipeline itself at 1024² with seed 42, and every failing edit then ran at 1024² with seed 42, so the edit's initial noise was the draw that generated the image. Changing only the seed fixes it:
A gradient-energy ratio near 1.0 means the over-sharpening signature is simply gone.
The mechanism is RNG-independent, which is worth stating because it makes the trap easy to hit without reusing a seed number. Reproducing it in a separate implementation (a Swift/MLX port of this pipeline): with that framework's own RNG the same edit is correct (20.9 dB, ratio 0.94), and injecting the torch seed-42 noise reproduces the failure exactly (13.05 dB, 2.88). What matters is only that the two draws coincide — same stream, seed and shape. The sharpest version: generating at 1024²/seed 42 in that port and then editing that output at 1024²/seed 42 also replays (13.73 dB, 2.97).
Correction. I claimed the ComfyUI cross-check used independent noise and that the failure was therefore noise-independent. That was wrong: ComfyUI's
prepare_noisedoestorch.manual_seed(seed)andtorch.randn(..., device="cpu"), the same streamrandn_tensoruses, so at the same shape and seed it drew the same noise. Measured against the diffusers draw: cosine 0.999999, max difference 0.015, which is exactly bf16 rounding. The ComfyUI run still shows this is not a diffusers implementation bug, and nothing more. Please disregard that claim.Everything else I reported collapses into the same explanation. 512/768/896/960 did not work because they are under a size threshold — a different target shape draws different noise. The mismatched probe I offered (768² reference, 1024² target) had a 1024² target, so it was replay as well. And a 1024² edit of a synthetic image the model never generated is clean, which is the control I should have run first.
So: nothing to fix in diffusers on my evidence. The only thing that might be worth a line in the docstring is that
output_resolutiondefaults to the size people most often generate at, which makes this easy to hit accidentally in a generate-then-edit workflow. Silently perturbing a user-suppliedgeneratorwould be worse than documenting it. For what it is worth, in the port I domain-separated the edit path's seed by a fixed constant, which is deterministic and removes the collision by construction — the previously-replaying case then edits correctly (17.27 dB, ratio 1.07).@peterc — mine is explained, but I do not think it covers yours: your text-to-image ran at 1312 with randomized seeds, so replay should not apply, and you saw it only on certain subjects. I would not want my report closing over your case.
sayakpaul commentedon Sep 22, 2026
Will keep the thread open but feel free to close if you think so.
xocialize commentedon Sep 22, 2026
I started to close this than I decided to give it some time in case we @peterc issue needes some more time here. @peterc let me know if you're ok closing this out as my issue is resolved with the above details.
peterc commentedon Sep 22, 2026
OK to close as it appears to be a separate issue from the noise replay issue entirely, so I will file again/elsewhere if I encounter the problem more generally again. Thanks!
jprsyt5 commentedon Sep 29, 2026
So apparently this also happens in Comfy. I opened an issue about it previously (#16607) thinking it was a resolution related or vae degradation, since it happened whenever I reused an edited output as the reference for another edit.
Changing the seed fixed it immediately. didn't know about "noise replay" until now.
So this just expected behavior for qwen image 2.1? I remember with qwen 2512 I could chain multiple edits with the same fixed seed across ksampler without running into this grain issue at all.