Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation

TL;DR

This work reframes text-to-image alignment as sampling from reward-weighted distributions, enabling plug-and-play guidance for both diffusion models and flow matching without fine-tuning. For diffusion models, it decomposes the guided score into the pre-trained score and a conditional reward-guidance term, and introduces a finetuning-free, regularized guidance network that estimates the required conditional expectation, achieving comparable one-step performance with at least a 60% reduction in computation. For flow matching, it derives an exact velocity guidance form and proposes a training-free estimator that improves generation quality without extra training costs. Empirically, the proposed methods outperform baselines on multiple metrics and demonstrate the practicality of one-step generation with strong alignment capabilities.

Abstract

Diffusion models and flow matching have demonstrated remarkable success in text-to-image generation. While many existing alignment methods primarily focus on fine-tuning pre-trained generative models to maximize a given reward function, these approaches require extensive computational resources and may not generalize well across different objectives. In this work, we propose a novel alignment framework by leveraging the underlying nature of the alignment problem -- sampling from reward-weighted distributions -- and show that it applies to both diffusion models (via score guidance) and flow matching models (via velocity guidance). The score function (velocity field) required for the reward-weighted distribution can be decomposed into the pre-trained score (velocity field) plus a conditional expectation of the reward. For the alignment on the diffusion model, we identify a fundamental challenge: the adversarial nature of the guidance term can introduce undesirable artifacts in the generated images. Therefore, we propose a finetuning-free framework that trains a guidance network to estimate the conditional expectation of the reward. We achieve comparable performance to finetuning-based models with one-step generation with at least a 60% reduction in computational cost. For the alignment on flow matching, we propose a training-free framework that improves the generation quality without additional computational cost.

Figures (3)

• Figure 1: Qualitative comparison with Vanilla SDXL, Diffusion-DPO, and SPO. Our method achieves better aesthetic quality and stronger alignment with the text prompt. Prompts are provided in the Appendix \ref{['ap:prompt']}.
• Figure 2: Illustration of the Adversarial Nature of Guidance. When the strength of the guidance is too small, there is little difference between the generated images with or without guidance. However, as the magnitude of the guidance increases (from left to right), undesirable artifacts become more pronounced. The prompt is "A 3D Rendering of a cockatoo wearing sunglasses. The sunglasses have a deep black frame with bright pink lenses. Fashion photography, volumetric lighting, CG rendering".
• Figure 3: Effectiveness of the proposed method for diffusion models: The results demonstrate that 2-step and 3-step generation significantly improve the quality of the generated images compared to one-step generation. While two vanilla guidance methods (Tweedie’s formula or directly backpropagation summarized in Section \ref{['sec:vanilla']}) fail to produce meaningful changes in the scene despite appropriate guidance strength, our method successfully achieves this enhancement. The prompt is "A photo of a frog holding an apple while smiling in the forest".

Theorems & Definitions (15)

• Theorem 3.1
• Lemma 3.2
• Theorem 4.1
• Theorem A.1
• proof
• Lemma A.2
• proof : Proof of Lemma \ref{['DSM-equ-guidance']}
• Lemma A.3
• proof : Proof of Lemma \ref{['thm:IS_guidance_conditional']}
• proof : Proof of Theorem \ref{['thm:fm']}