Fibo Edit: multi-reference conditioning and batching by ShivamShrirao · Pull Request #14566 · huggingface/diffusers

ShivamShrirao · 2026-08-23T10:06:33Z

What does this PR do?

Adds multi-reference image conditioning to BriaFiboEditPipeline and fixes the two remaining items from the bria_fibo review issue.

Fixes the remaining items (issues 2 and 5) of #13618, issues 1/3/4/6 were fixed by #13981.

Multi-reference conditioning

image now accepts a PIL.Image.Image or a list of them. A list means multiple references, not a batch:

Each reference is VAE-encoded at its own aspect ratio (area capped at 1024², dims rounded to /16), packed, and appended to the sequence as context tokens.
Reference i is placed on RoPE time plane i (generated tokens stay on plane 0).
Output resolution defaults to the first reference's size; explicit height/width still override it.
Masks remain supported with exactly one reference.

Batching

Prompt lists are now accepted (references repeat across the batch).
num_images_per_prompt > 1 now works, upstream crashed at a latent reshape because reference latents were never duplicated (fixes issue 5 of bria_fibo model/pipeline review #13618). test_num_images_per_prompt is un-skipped and passes.

Other fixes

Custom timesteps are now honored in both BriaFiboPipeline and BriaFiboEditPipeline, they were accepted and documented but hard-coded to None in the retrieve_timesteps call (fixes issue 2 of bria_fibo model/pipeline review #13618).
The attention mask is now a bool key-padding mask (batch, 1, 1, seq) instead of a float outer-product matrix, and is dropped entirely when nothing is padded, this keeps backends without mask support (flash-attn 2/3) usable at guidance_scale <= 1.
BriaFiboTransformer2DModel declares _repeated_blocks for regional compilation.
do_patching=True previously produced mismatched token widths between generated and reference latents (references were always packed unpatched); reference packing now follows do_patching.

Intentional behavior changes (please review)

torch.Tensor image input now raises. References are PIL-only, following the Flux2 / QwenImage Edit Plus convention. Tensor input only became functional recently (fix(bria_fibo): fix guidance_embeds, prompt_embeds, tensor-image and multi-image crashes #13981 fixed a crash); np.ndarray was already rejected.
List semantics changed: a list of images previously went into preprocess as a batch; it now means multiple references. image=[img] behaves identically to before.
A single reference is no longer resized to the output resolution, it is encoded at its own aspect ratio like any other reference. With default height/width (taken from the reference) behavior is equivalent; it only diverges when an explicit output size differs from the reference.
Reference-latent normalization runs in the latent's dtype (repo convention, as in Flux2/Wan/Qwen). Verified against fp32 normalization on real weights: MAE 0.95/255, visually identical.

Validation

Generation verified end-to-end on real Fibo-Edit-1.5 weights (single and multi-reference, batched prompts).
Reference indexing/RoPE layout verified against the internal training code (reference i → time id i, per-reference variable resolution, target-first sequence order).
Fast suites: all bria_fibo + bria_fibo_edit tests pass. The pre-existing test_inference_batch_single_identical failure in the base bria_fibo suite reproduces on current main without this PR, not introduced here.
New tests: distinct RoPE time planes per reference, batched prompts × multiple references, mask-requires-single-reference.
make style, make fix-copies clean.

Docs: added a "Multiple reference images" section to bria_fibo_edit.md with a runnable two-reference example; fixed and extended EXAMPLE_DOC_STRING.

Self-review notes (final round)

Rubric: .ai/review-rules.md (+ .ai/AGENTS.md, .ai/pipelines.md, .ai/testing.md, .ai/models.md, model-integration pitfalls). Scope: full diff vs upstream base.

Verdict: READY

No blocking findings. Reference preparation is a single public lifecycle method (prepare_reference_latents) called from __call__, one code path handles one or N references, and tests assert on public API and end-to-end outputs only.

Checks: full bria_fibo + bria_fibo_edit fast suites pass (aside from the pre-existing failure noted below), make style clean, check_copies clean.

Behavior changes (disclosed intentionally)

torch.Tensor image input now raises (was supported upstream since fix(bria_fibo): fix guidance_embeds, prompt_embeds, tensor-image and multi-image crashes #13981). References are PIL-only, following the Flux2 / QwenImage Edit Plus convention. np.ndarray was already rejected upstream, unchanged.
List semantics changed: upstream fed a list into preprocess as a batch; a list now means multiple references. image=[img] behaves identically to before; image=[a, b] is a two-reference edit.
Single reference is no longer resized to the output resolution, it is encoded at its own aspect ratio (1024² area cap), same as any other reference; output resolution defaults from the first reference.
Prompt lists accepted (batched prompts); reference latents repeat across the batch.
Custom timesteps are now honored in both pipelines (fixes issue 2 of bria_fibo model/pipeline review #13618): they were accepted and documented but hard-coded to None in the retrieve_timesteps call, silently falling back to the default schedule.
num_images_per_prompt > 1 now works in the edit pipeline (fixes issue 5 of bria_fibo model/pipeline review #13618): upstream crashed at a latent reshape because reference latents were never duplicated. Everything now keys off the encoded prompt batch (batch_size * num_images_per_prompt); test_num_images_per_prompt is un-skipped and passes.

Left for the reviewer

_vae_safe_dims / _vae_safe_size carry base_resolution=1024 / multiple=16 defaults no caller overrides, named constants vs. unused knobs is a judgment call.
Pre-existing test_inference_batch_single_identical failure in the base bria_fibo pipeline reproduces on main without this PR, not introduced here.

Dead-code analysis (advisory)

Traced __call__ end to end: paste_mask_on_image, is_valid_mask, get_mask_size, and both do_patching branches are all reachable. No likely-dead code beyond the default params noted above.

Docs

docs/source/en/api/pipelines/bria_fibo_edit.md gained a "Multiple reference images" section (semantics + runnable two-reference example); EXAMPLE_DOC_STRING fixed (imports) and extended with a multi-reference call.

Before submitting

Who can review?

@yiyixuxu @sayakpaul

…pt flow

…base/turbo ids briaai/FIBO-edit-prompt-to-JSON is retired in favor of briaai/FIBO-edit-gemini-prompt-to-JSON, which handles multiple reference images and masks. Also corrects the checkpoint ids to their canonical briaai/Fibo-Edit-1.5-base and briaai/Fibo-Edit-1.5-turbo spellings.

yiyixuxu

thanks for the PR! i left some small comments

yiyixuxu · 2026-08-24T19:10:40Z

+    return tuple(max(multiple, int(round(side * scale / multiple)) * multiple) for side in (width, height))
+
+
+def _vae_safe_size(image, base_resolution=1024, multiple=16):


can we in-line this function?

the image_processor.preprocess change below does the resize, so this helper is no longer needed.

yiyixuxu · 2026-08-24T19:12:57Z

+    return image if target == image.size else image.resize(target, Image.LANCZOS)
+
+
+def _as_reference_images(image):


can we put this code inside call directly?

Moved into call. It had a second call site in check_inputs, so rather than duplicate the validation I normalize once at the top of call and pass the list down.

yiyixuxu · 2026-08-24T19:17:54Z

+        """VAE-encode one PIL reference at its own size and pack it as an edit-context token stream."""
+        vae_dtype = next(self.vae.parameters()).dtype
+        image = _vae_safe_size(image.convert("RGB"))
+        pixels = torch.from_numpy(np.array(image)).permute(2, 0, 1).unsqueeze(0)


can we use the self.image_processor.preprocess(...)?

Done. I pass explicit height/width from _vae_safe_dims so the multiple-of-16 rounding and the 1024²-pixel cap are preserved. Verified the output is identical to the manual path.

yiyixuxu · 2026-08-24T19:22:29Z

@bot /style

github-actions · 2026-08-24T19:23:55Z

Style fix is beginning .... View the workflow run here.

HuggingFaceDocBuilderDev · 2026-08-24T19:28:23Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

- collapse the mask branch to `if not attention_mask.all():` in both Fibo pipelines - encode references via `self.image_processor.preprocess(...)`, dropping the manual numpy normalization and the `_vae_safe_size` helper - normalize `image` into a reference list once in `__call__`, pass it to `check_inputs`

yiyixuxu · 2026-08-24T20:22:50Z

@ShivamShrirao
can you run make style? once CI pass we will merge:)

ShivamShrirao · 2026-08-24T20:31:36Z

@yiyixuxu done!

Fibo Edit: multi-reference conditioning, batching, timesteps

479af16

github-actions Bot added documentation Improvements or additions to documentation models tests pipelines size/L PR with diff > 200 LOC labels Aug 23, 2026

sayakpaul requested a review from dg845 August 23, 2026 14:33

docs: simplify Fibo Edit page, fix EXAMPLE_DOC_STRING to the VLM prom…

91796cd

…pt flow

ShivamShrirao force-pushed the fibo-edit-multi-reference branch from 07d3dd6 to 91796cd Compare August 23, 2026 14:35

ShivamShrirao added 2 commits August 24, 2026 17:43

Merge remote-tracking branch 'hf/main' into fibo-edit-multi-reference

ce93a61

yiyixuxu approved these changes Aug 24, 2026

View reviewed changes

ShivamShrirao added 2 commits August 24, 2026 20:27

make style

a447dfd

make style

cf6a8cd

Navigation Menu

Uh oh!

Fibo Edit: multi-reference conditioning and batching - #14566

Fibo Edit: multi-reference conditioning and batching#14566
ShivamShrirao wants to merge 7 commits into
huggingface:mainfrom
Bria-AI:fibo-edit-multi-reference

ShivamShrirao commented Aug 23, 2026 •

edited

Loading

Uh oh!

yiyixuxu left a comment

Uh oh!

Uh oh!

yiyixuxu Aug 24, 2026

Uh oh!

ShivamShrirao Aug 24, 2026

Uh oh!

yiyixuxu Aug 24, 2026

Uh oh!

ShivamShrirao Aug 24, 2026

Uh oh!

yiyixuxu Aug 24, 2026

Uh oh!

ShivamShrirao Aug 24, 2026

Uh oh!

yiyixuxu commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Aug 24, 2026

Uh oh!

yiyixuxu commented Aug 24, 2026

Uh oh!

ShivamShrirao commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return tuple(max(multiple, int(round(side * scale / multiple)) * multiple) for side in (width, height))


		def _vae_safe_size(image, base_resolution=1024, multiple=16):

		return image if target == image.size else image.resize(target, Image.LANCZOS)


		def _as_reference_images(image):

Uh oh!

Conversation

ShivamShrirao commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Multi-reference conditioning

Batching

Other fixes

Intentional behavior changes (please review)

Validation

Before submitting

Who can review?

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ShivamShrirao Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

yiyixuxu Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ShivamShrirao Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

yiyixuxu Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ShivamShrirao Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Aug 24, 2026

Uh oh!

yiyixuxu commented Aug 24, 2026

Uh oh!

ShivamShrirao commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ShivamShrirao commented Aug 23, 2026 •

edited

Loading