Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discovering Latent Graphs with GFlowNets for Diverse Conditional Image Generation

Bailey Trang, Parham Saremi, Alan Q. Wang, Fangrui Huang, Zahra TehraniNasab, Amar Kumar, Tal Arbel, Li Fei-Fei, Ehsan Adeli

TL;DR

<3-5 sentence high-level summary> Rainbow tackles inherent uncertainty in conditional image generation by decomposing a given condition into diverse latent representations using a latent graph learned via Generative Flow Networks (GFlowNets). By sampling multiple diverse trajectories over this latent graph, Rainbow produces a set of condition representations which are decoded and fed into a pretrained latent diffusion model to generate multiple plausible outputs for a single prompt. The approach is compatible with any pretrained conditional generator and demonstrates improved diversity and fidelity across natural and medical imaging tasks, while enabling interpretable latent-graph analysis and useful downstream capabilities like editing and counterfactual synthesis. Overall, Rainbow provides a flexible, scalable mechanism to realize diverse, high-quality conditional generation by coupling GFlowNets with latent diffusion in a graph-informed latent space.

Abstract

Capturing diversity is crucial in conditional and prompt-based image generation, particularly when conditions contain uncertainty that can lead to multiple plausible outputs. To generate diverse images reflecting this diversity, traditional methods often modify random seeds, making it difficult to discern meaningful differences between samples, or diversify the input prompt, which is limited in verbally interpretable diversity. We propose Rainbow, a novel conditional image generation framework, applicable to any pretrained conditional generative model, that addresses inherent condition/prompt uncertainty and generates diverse plausible images. Rainbow is based on a simple yet effective idea: decomposing the input condition into diverse latent representations, each capturing an aspect of the uncertainty and generating a distinct image. First, we integrate a latent graph, parameterized by Generative Flow Networks (GFlowNets), into the prompt representation computation. Second, leveraging GFlowNets' advanced graph sampling capabilities to capture uncertainty and output diverse trajectories over the graph, we produce multiple trajectories that collectively represent the input condition, leading to diverse condition representations and corresponding output images. Evaluations on natural image and medical image datasets demonstrate Rainbow's improvement in both diversity and fidelity across image synthesis, image generation, and counterfactual generation tasks.

Discovering Latent Graphs with GFlowNets for Diverse Conditional Image Generation

TL;DR

<3-5 sentence high-level summary> Rainbow tackles inherent uncertainty in conditional image generation by decomposing a given condition into diverse latent representations using a latent graph learned via Generative Flow Networks (GFlowNets). By sampling multiple diverse trajectories over this latent graph, Rainbow produces a set of condition representations which are decoded and fed into a pretrained latent diffusion model to generate multiple plausible outputs for a single prompt. The approach is compatible with any pretrained conditional generator and demonstrates improved diversity and fidelity across natural and medical imaging tasks, while enabling interpretable latent-graph analysis and useful downstream capabilities like editing and counterfactual synthesis. Overall, Rainbow provides a flexible, scalable mechanism to realize diverse, high-quality conditional generation by coupling GFlowNets with latent diffusion in a graph-informed latent space.

Abstract

Capturing diversity is crucial in conditional and prompt-based image generation, particularly when conditions contain uncertainty that can lead to multiple plausible outputs. To generate diverse images reflecting this diversity, traditional methods often modify random seeds, making it difficult to discern meaningful differences between samples, or diversify the input prompt, which is limited in verbally interpretable diversity. We propose Rainbow, a novel conditional image generation framework, applicable to any pretrained conditional generative model, that addresses inherent condition/prompt uncertainty and generates diverse plausible images. Rainbow is based on a simple yet effective idea: decomposing the input condition into diverse latent representations, each capturing an aspect of the uncertainty and generating a distinct image. First, we integrate a latent graph, parameterized by Generative Flow Networks (GFlowNets), into the prompt representation computation. Second, leveraging GFlowNets' advanced graph sampling capabilities to capture uncertainty and output diverse trajectories over the graph, we produce multiple trajectories that collectively represent the input condition, leading to diverse condition representations and corresponding output images. Evaluations on natural image and medical image datasets demonstrate Rainbow's improvement in both diversity and fidelity across image synthesis, image generation, and counterfactual generation tasks.
Paper Structure (38 sections, 10 equations, 30 figures, 4 tables)

This paper contains 38 sections, 10 equations, 30 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Generative Flow Networks (GFlowNets)
  4. Detailed Balance Objective
  5. Capturing Diversity with GFlowNets
  6. Latent Diffusion Models
  7. Rainbow
  8. Rainbow's Inference Pipeline
  9. Rainbow's Training Details
  10. Experiment
  11. Experiment Setup
  12. Experiment Results
  13. Diverse Images Generation Results
  14. Latent Graph Interpretation
  15. Performance on Downstream Tasks
  16. ...and 23 more sections

Figures (30)

  • Figure 1: Rainbow operates by transforming an input condition into diverse images. Initially, it employs the pretrained condition encoder to derive an initial representation of the input condition (which contains uncertainty about locations or objects with the given prompt in this example). Then, a graphs generator produces multiple trajectories over a graph that reflect the input condition. These graphs are encoded into new latent condition representations. New condition representations and a latent noisy image are processed through the Latent Diffusion Model to acquire denoised image latents, which are subsequently decoded into diverse output images.
  • Figure 2: Comparison of multiple images generated by baselines and Rainbow. The baseline methods tend to produce images with repetitive layouts and primarily drawing art styles, failing to capture the uncertainty of "season". In contrast, Rainbow generates a variety of sunset scenes, showcasing diverse light levels, grass colors, and effectively capturing different seasons.
  • Figure 3: Images generated by Rainbow seasonal edges with the base prompt "Sunset scene with mountain". Most objects and layouts are consistent between the images, with noticeable season-specific details in the second image, such as spring flowers and winter snow.
  • Figure 4: Quantitative analysis on diversity and image quality of SD-based Rainbow. Rainbow consistently outperforms SD baselines in diversity with higher Vendi Score (VS) across domains and image quality with higher Inception Score (IS) in natural images and FID$^{-1}$ in chest X-rays.
  • Figure 5: Comparison of MRI image generations for 65-year-old male individual. Compared to actual samples from males aged 63-65, Rainbow captures greater diversity in details like ventricle sizes, while the baseline LDM generates images with less variation.
  • ...and 25 more figures