Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models

TL;DR

The paper addresses text–image misalignment in diffusion-based generation by identifying a training–inference mismatch: prompt-conditioned noises during training occupy a prompt-specific subspace, while inference samples come from a prompt-agnostic Gaussian prior. It introduces a noise projector that, guided by the full prompt embedding, refines the initial noise before denoising, aligning its distribution with SD training without modifying the backbone. A reinforcement-learning–style framework trains the projector using token-level rewards predicted by a compact reward model that mimics a vision–language model, enabling dense feedback via quasi-direct preference optimization. The approach yields improved prompt–image alignment in both single- and multi-prompt settings, with lower inference cost than multi-sample methods and without the need for reference images. These results demonstrate a practical, scalable path to robust text-to-image alignment in diffusion models.

Abstract

In text-to-image generation, different initial noises induce distinct denoising paths with a pretrained Stable Diffusion (SD) model. While this pattern could output diverse images, some of them may fail to align well with the prompt. Existing methods alleviate this issue either by altering the denoising dynamics or by drawing multiple noises and conducting post-selection. In this paper, we attribute the misalignment to a training-inference mismatch: during training, prompt-conditioned noises lie in a prompt-specific subset of the latent space, whereas at inference the noise is drawn from a prompt-agnostic Gaussian prior. To close this gap, we propose a noise projector that applies text-conditioned refinement to the initial noise before denoising. Conditioned on the prompt embedding, it maps the noise to a prompt-aware counterpart that better matches the distribution observed during SD training, without modifying the SD model. Our framework consists of these steps: we first sample some noises and obtain token-level feedback for their corresponding images from a vision-language model (VLM), then distill these signals into a reward model, and finally optimize the noise projector via a quasi-direct preference optimization. Our design has two benefits: (i) it requires no reference images or handcrafted priors, and (ii) it incurs small inference cost, replacing multi-sample selection with a single forward pass. Extensive experiments further show that our prompt-aware noise projection improves text-image alignment across diverse prompts.

Figures (5)

• Figure 1: The comparison among several denoising patterns. Green path denotes normal denoising with a pretrained model, which has the risk of inducing text-image misalignment problem. Purple path reveals that optimization-based methods in essence modifies the ODE sampler locally. Red path denotes denoising from a better noise, which could be regarded as sampled from a prompt-conditioning distribution instead of normal Gaussian.
• Figure 2: The architecture of our model. Both the noise projector and the reward model share the same backbone with cross-attention, MoE, and UNet components. At the output stage, the reward model attaches a classification head, while the noise projector integrates the encoder of a VAE.
• Figure 3: The overall framework of our method, which consists of four stages: (a) data preparation for training the reward model (Section \ref{['subsubsection: prepare data for reward model']}), (b) reward model training with the collected data (Section \ref{['subsubsection: align reward model']}), (c) pretraining the noise projector (Section \ref{['subsubsection: pretrain noise projector']}), and (d) final quasi-DPO training (Section \ref{['subsubsection: final training']}).
• Figure 4: Qualitative comparison of generated images. For complex prompts that often induce text–image misalignment, our noise projector effectively refines the noise to yield well-aligned generations. In cases where the original SDXL already produces satisfactory results (i.e., the rightmost column), our noise projector leaves the structure largely unchanged, preserving the same well-aligned layout as the original.
• Figure 5: The innerly-computed FID and IS comparison with a single prompt.