Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis

TL;DR

This work proposes an implementation of FG-DMs by adapting a pre-trained Stable Diffusion model to implement all FG-DM factors, and shows that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability.

Abstract

Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text in images, and meeting constraints like object locations and pose. For fine-grained editing and manipulation, they also require fine-grained semantic or instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions at inference, this is time consuming and does not scale to complex scenes. To overcome these limitations, this work introduces a new family of \textit{Factor Graph Diffusion Models} (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, semi-automated fine-grained editing, text-based editing of conditions with noise inversion, explainability at intermediate levels, ability to produce labeled datasets for the training of downstream models such as segmentation or depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose an implementation of FG-DMs by adapting a pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image.

Figures (28)

• Figure 1: Comparison of FG-DM (bottom) against Stable Diffusion (top) for sampling images with high object recall by modeling the joint distribution of images and conditioning variables. FG-DM supports creative, controllable, interpretable and faster (4x) image synthesis than Stable Diffusion to achieve the desired object recall. Note that the conditions $\mathbf{y^1}$ or $\mathbf{y^2}$ can be null due to classifier-free guidance training.
• Figure 1: User study on the qualitative preference of images/condition pairs generated by the FG-DM and SD+CEM, using 10 unique human evaluators. A. denotes (prompt) Adherence and Q. denotes Quality.
• Figure 2: FG-DM-based controllable image generation via editing segmentation, depth and sketch maps. Top: generated conditions and images. Bottom: edited ones. Note that only the segmentation map is edited, pose and images are conditionally generated given edited map.
• Figure 3: Synthesized segmentation/depth/sketch/normal maps and corresponding images by an FG-DM adapted from SD using COCO. The FG-DM generalizes to prompts beyond this dataset such as porcupine, chimp and other creative prompts shown.
• Figure 3: Object recall statistics for sampling FG-DM with different seeds and timesteps on the ADE20K validation set prompts.