Saliency Guided Optimization of Diffusion Latents

TL;DR

The paper addresses misalignment in diffusion-based text-to-image generation caused by uniform global optimization. It introduces SGOOL, a saliency-guided latent optimization framework that uses a saliency detector to identify perceptually important regions and jointly optimizes global image and salient parts via a CLIP-based loss, all without retraining thanks to an invertible diffusion process that enables memory-efficient backpropagation. The method is plug-and-play and demonstrates notable improvements in both image quality and prompt alignment on multiple datasets, with quantitative gains of about $+3.05$ in CLIP score and $+0.0029$ in HPS, along with qualitative enhancements. These results suggest a practical path to more faithful and detail-rich image generation by incorporating human visual saliency into diffusion latent optimization.$

Abstract

With the rapid advances in diffusion models, generating decent images from text prompts is no longer challenging. The key to text-to-image generation is how to optimize the results of a text-to-image generation model so that they can be better aligned with human intentions or prompts. Existing optimization methods commonly treat the entire image uniformly and conduct global optimization. These methods overlook the fact that when viewing an image, the human visual system naturally prioritizes attention toward salient areas, often neglecting less or non-salient regions. That is, humans are likely to neglect optimizations in non-salient areas. Consequently, although model retaining is conducted under the guidance of additional large and multimodality models, existing methods, which perform uniform optimizations, yield sub-optimal results. To address this alignment challenge effectively and efficiently, we propose Saliency Guided Optimization Of Diffusion Latents (SGOOL). We first employ a saliency detector to mimic the human visual attention system and mark out the salient regions. To avoid retraining an additional model, our method directly optimizes the diffusion latents. Besides, SGOOL utilizes an invertible diffusion process and endows it with the merits of constant memory implementation. Hence, our method becomes a parameter-efficient and plug-and-play fine-tuning method. Extensive experiments have been done with several metrics and human evaluation. Experimental results demonstrate the superiority of SGOOL in image quality and prompt alignment.

Figures (7)

• Figure 1: Traditional methods vs. SGOOL. Given a prompt: "A giant white furry monster stomps into a city, camera looking up from street view.". SGOOL generates an image with higher quality and more details.
• Figure 2: The overall pipeline of SGOOL. SGOOL includes image generation, saliency detection, and optimizing steps. SGOOL obtains the saliency parts through the saliency detection model $\Gamma$. Then, both the global image and saliency parts are considered for optimization to improve the quality of the image.
• Figure 3: Box plots for CLIP score and HPS.
• Figure 4: Bar plots for CLIP score and HPS.
• Figure 5: Examples of images generated by (a) vanilla Stable Diffusion (SD), (b) Stable Diffusion with CLIP guidance (Baseline), (c) DOODL, and (f) SGOOL (Ours) from the same random seed.