Improving Long-Text Alignment for Text-to-Image Diffusion Models

TL;DR

This work tackles the challenge of aligning text-to-image diffusion models with long, detailed prompts by introducing LongAlign, a framework combining segment-level text encoding with decomposed preference optimization. Segment-level encoding mitigates CLIP's token limits by processing text in segments and carefully merging embeddings, while segment-level preferences generate per-segment scores and an aggregate Denscore. The core innovation is decomposing CLIP-based preference scores into text-relevant and text-irrelevant components, then applying a gradient reweighting strategy to reduce overfitting during reward fine-tuning. Empirically, LongAlign enables a 512×512 Stable Diffusion model to surpass stronger foundation models in long-text alignment after about 20 hours of training, and the approach scales to higher resolutions like SDXL, demonstrating practical impact for complex, long prompts.

Abstract

The rapid advancement of text-to-image (T2I) diffusion models has enabled them to generate unprecedented results from given texts. However, as text inputs become longer, existing encoding methods like CLIP face limitations, and aligning the generated images with long texts becomes challenging. To tackle these issues, we propose LongAlign, which includes a segment-level encoding method for processing long texts and a decomposed preference optimization method for effective alignment training. For segment-level encoding, long texts are divided into multiple segments and processed separately. This method overcomes the maximum input length limits of pretrained encoding models. For preference optimization, we provide decomposed CLIP-based preference models to fine-tune diffusion models. Specifically, to utilize CLIP-based preference models for T2I alignment, we delve into their scoring mechanisms and find that the preference scores can be decomposed into two components: a text-relevant part that measures T2I alignment and a text-irrelevant part that assesses other visual aspects of human preference. Additionally, we find that the text-irrelevant part contributes to a common overfitting problem during fine-tuning. To address this, we propose a reweighting strategy that assigns different weights to these two components, thereby reducing overfitting and enhancing alignment. After fine-tuning $512 \times 512$ Stable Diffusion (SD) v1.5 for about 20 hours using our method, the fine-tuned SD outperforms stronger foundation models in T2I alignment, such as PixArt-$α$ and Kandinsky v2.2. The code is available at https://github.com/luping-liu/LongAlign.

Figures (20)

• Figure 1: Generation results of our long Stable Diffusion and baselines. We highlight three key facts for each prompt and provide the evaluation results at the end. In each evaluation line, the four group results are arranged in order of model presentation, with S representing SD-1.5, and so on. Additionally, three $\checkmark$ or $\times$ maintain the order of the key facts corresponding to each prompt.
• Figure 2: (a) Schematic results for text embeddings. (b) Statistics of the projection scalar $\eta$ for three models. (c) The relationship between the original score and the two scores after decomposition using our Denscore. In the three score tables, the diagonal represents the scores for paired data, while the off-diagonal positions indicate the scores for unpaired data.
• Figure 3: Retrieval results with low or high text-irrelevant scores, using three CLIP-based models.
• Figure 4: FID and Denscore results for diffusion models with different text encodings.
• Figure 5: FID and Denscore results for diffusion models using different gradient reweighting factors.