An Intermediate Fusion ViT Enables Efficient Text-Image Alignment in Diffusion Models

TL;DR

It is discovered that compared to the widely used early fusion of conditioning text in a pretrained image feature space, a specially designed intermediate fusion can boost text-to-image alignment with improved generation quality and improve training and inference efficiency by reducing low-rank text-to-image attention calculations.

Abstract

Diffusion models have been widely used for conditional data cross-modal generation tasks such as text-to-image and text-to-video. However, state-of-the-art models still fail to align the generated visual concepts with high-level semantics in a language such as object count, spatial relationship, etc. We approach this problem from a multimodal data fusion perspective and investigate how different fusion strategies can affect vision-language alignment. We discover that compared to the widely used early fusion of conditioning text in a pretrained image feature space, a specially designed intermediate fusion can: (i) boost text-to-image alignment with improved generation quality and (ii) improve training and inference efficiency by reducing low-rank text-to-image attention calculations. We perform experiments using a text-to-image generation task on the MS-COCO dataset. We compare our intermediate fusion mechanism with the classic early fusion mechanism on two common conditioning methods on a U-shaped ViT backbone. Our intermediate fusion model achieves a higher CLIP Score and lower FID, with 20% reduced FLOPs, and 50% increased training speed compared to a strong U-ViT baseline with an early fusion.

Figures (12)

• Figure 1: Text-to-image data generation using our intermediate fusion mechanism and the classic early fusion mechanism: images are generated based on a text input using the corresponding mechanism in each row. The input text contains a type of high-level semantics in each column. We observe that our mechanism generates better quality samples that align better with the text query while also being more efficient.
• Figure 2: The two conditioning methods (top row) and two fusion methods (bottom row) in our experiments. The baseline is a 13-layer ViT model with the same skip-connection mechanisms introduced in U-ViT. For fusion comparison, we carefully designed $N_\text{joint}$ to ensure same depth and number of parameters for the image branch. In the experiments we use $N_\text{image} = 4$ and $N_\text{text} = 1$. For conditioning comparison, we switch the attention type and skip connections with other blocks intact.
• Figure 3: Evaluation during training and FID-30K vs CLIP Score at different CFG scales. Intermediate fusion settings show improved generation quality and text-image alignment compared to their early fusion counterparts. CLIP Score is measured on 30K pairs using CLIP-ViT-L-14.
• Figure 4: Comparison between the best intermediate fusion model(bottom) and the baseline model(top) across 12 different prompts. (Best viewed when zoomed-in.)
• Figure 5: Human evaluation on object count. Lighter colors represent early fusion, while darker colors represent intermediate fusion. The left four figures are the average error given different ground truth counts, where x-axis is the ground truth. Each figure corresponds to an object. The right top figure is the average error across all counts for different objects. The bottom right figure is the average percentage of exact matches for each object. The plots indicated lower average count errors and higher matching counts of intermediate fusion.