RDiffusion-AMV

Attention-Based Prompting and Verifier-Based Selection for Layout-to-Image Generation

Overview

RDiffusion-AMV is a layout-to-image generation framework built on top of GLIGEN and Stable Diffusion.
The project improves spatial grounding and object consistency using:

• Spatial-aware conditioning
• Depth-aware embeddings
• Relation-aware embeddings
• Attention masking
• Region-aware prompt augmentation
• Verifier-based candidate selection

The system generates multiple candidate images and automatically selects the best layout-faithful output using a weighted verifier scoring pipeline.

---

Architecture

---

Key Features

Enhanced Conditioning

• Spatial encoding using bounding boxes
• Depth-aware conditioning using MiDaS depth priors
• Relation-aware scene encoding
• Unified scene embedding generation

Region-Aware Prompting (AM)

• Prompt augmentation with spatial instructions
• Better object placement guidance
• Improved grounding consistency

Frozen GLIGEN Backbone

• Uses pretrained GLIGEN diffusion backbone
• No retraining of Stable Diffusion backbone
• Lightweight conditioning extensions

Multi-Candidate Generation

• Generates multiple candidates using different seeds
• Improves diversity and selection quality

Verifier-Based Selection (AMV)

Candidates are evaluated using:

• IoU (Spatial Alignment)
• OCS (Object-Centric Spatial Consistency)
• CLIP Similarity
• RSS (Relation Satisfaction Score)
• Boundary Sharpness

The best candidate is selected using weighted verifier scoring.

Interactive Frontend

• Gradio-based frontend
• User-defined prompts
• Bounding-box control
• Depth-aware object placement

---

Repository Structure

RDiffusion-AMV/
│
├── RDiffusion_AMV_Final_GitHub_Notebook.ipynb
├── README.md
├── requirements.txt
│
├── assets/
│   ├── architecture.png
│   └── sample_outputs/
│
├── outputs/
├── checkpoints/
└── data/

---

# Results Summary

| Model Variant                          | Images Generated | Scenes |
| -------------------------------------- | ---------------: | -----: |
| SD Baseline                            |              100 |      0 |
| GLIGEN Vanilla                         |              804 |    100 |
| GLIGEN Enriched                        |              796 |    100 |
| GLIGEN Refined Bounding Box            |              800 |    100 |
| RDiffusion Trained                     |              264 |     33 |
| RDiffusion Attention Masked            |              256 |     32 |
| RDiffusion Attention Masked + Verifier |               32 |     32 |
| RDiffusion True Attention Masked       |              202 |     26 |

---

# Setup Instructions

## Environment

Recommended:

* Google Colab A100 GPU
* Python 3.10+
* CUDA-enabled environment

---

## Install Dependencies

```bash
pip install -r requirements.txt

---

External Assets Required

Large datasets and checkpoints are not uploaded due to GitHub size limitations.

You must manually place the following in your Google Drive:

/content/drive/MyDrive/RDiffusion/
│
├── GLIGEN/
├── checkpoints/
├── data/
├── outputs/

Required assets include:

• GLIGEN checkpoints
• Stable Diffusion weights
• Trained tokenizer checkpoints
• COCO/VG datasets
• Preprocessed scene JSON files

---

How to Run

Step 1 — Open Notebook

Open:

RDiffusion_AMV_Final_GitHub_Notebook.ipynb

in Google Colab.

---

Step 2 — Mount Google Drive

from google.colab import drive
drive.mount('/content/drive')

---

Step 3 — Run Setup Cells

Run:

• imports
• GLIGEN patching
• tokenizer definitions
• preprocessing utilities

---

Step 4 — Run Generation Cells

The notebook supports:

• Stable Diffusion baseline generation
• GLIGEN generation
• RDiffusion generation
• Attention masked generation
• Verifier-based selection

---

Step 5 — Launch Frontend

The notebook includes a Gradio frontend for interactive generation.

Features:

• Prompt input
• Bounding box controls
• Depth-aware conditioning
• Multi-candidate generation

---

Frontend Fix

If Gradio throws an InvalidPathError, use:

demo.launch(
    share=True,
    debug=True,
    allowed_paths=[
        "/content/drive/MyDrive/RDiffusion/outputs/frontend_rdiffusion_amv"
    ]
)

---

Reproducing Results

To reproduce results:

1. Prepare datasets and checkpoints
2. Run preprocessing cells
3. Run tokenizer loading/setup
4. Run generation pipeline
5. Run evaluation pipeline
6. Run verifier-based selection

Generated outputs will be saved to:

/content/drive/MyDrive/RDiffusion/outputs/

---

Notes

• Existing generated outputs are automatically skipped during reruns.
• The notebook supports interrupted runtime recovery.
• Designed primarily for Google Colab execution.

---

Attribution

This project builds upon and adapts components from:

• GLIGEN
• Stable Diffusion
• Hugging Face Transformers
• OpenAI CLIP

Official GLIGEN repository: https://github.com/gligen/GLIGEN