Boosting Generative Image Modeling via Joint Image-Feature Synthesis

TL;DR

This work introduces ReDi, a joint latent-semantic diffusion framework that models VAE latents and DINOv2 semantic features within a single diffusion process. By eliminating distillation objectives and adding Representation Guidance for inference, ReDi achieves markedly higher image quality and dramatically faster convergence on ImageNet compared to prior approaches like REPA. The method is applicable to both Diffusion Transformer (DiT) and Diffusion Transformer-based SiT backbones, and is effective in both conditional and unconditional generation, including improvements in unconditional FID when guidance is used. Through token fusion strategies and PCA-based dimensionality reduction, ReDi maintains computational efficiency while delivering strong, scalable gains, establishing a representation-aware direction for generative modeling.

Abstract

Latent diffusion models (LDMs) dominate high-quality image generation, yet integrating representation learning with generative modeling remains a challenge. We introduce a novel generative image modeling framework that seamlessly bridges this gap by leveraging a diffusion model to jointly model low-level image latents (from a variational autoencoder) and high-level semantic features (from a pretrained self-supervised encoder like DINO). Our latent-semantic diffusion approach learns to generate coherent image-feature pairs from pure noise, significantly enhancing both generative quality and training efficiency, all while requiring only minimal modifications to standard Diffusion Transformer architectures. By eliminating the need for complex distillation objectives, our unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance, which leverages learned semantics to steer and refine image generation. Evaluated in both conditional and unconditional settings, our method delivers substantial improvements in image quality and training convergence speed, establishing a new direction for representation-aware generative modeling. Project page and code: https://representationdiffusion.github.io

Figures (16)

• Figure 1: ReDi: Our generative image modeling framework bridges the gap between generative modeling and representation learning by leveraging a diffusion model that jointly captures low-level image details (via VAE latents) and high-level semantic features (via DINOv2). Trained to generate coherent image–feature pairs from pure noise, this unified latent-semantic dual-space diffusion approach significantly boosts both generative quality and training convergence speed.
• Figure 2: Accelerated Training. Generative performance curves on Imagenet $256 \times 256$ without Classifier-Free Guidance. Left: Our ReDi accelerates convergence of DiT-XL/2 and SiT-XL/2 by approximately $\times 23$. Right:ReDi converges $\times 6$ faster than REPA. When applied on top of REPA delivers a $\times 11$ speed-up.
• Figure 3: Given an input image, the VAE latent and the principal components of DINOv2 are extracted. Both modalities are noised and fused into a joint token sequence, given as input to DiT or SiT.
• Figure 4: An illustration of our proposed token fusion approaches: (a) The tokens of the $\texttt{VAE}$ latents and the $\texttt{DINOv2}$ are merged channel-wise, (b) The tokens are concatenated along the sequence dimension.
• Figure 5: Selected samples from our SiT-XL/2 w/ ReDi model trained on ImageNet $256\times256$. Images and visual representations are jointly generated by our model. We use Classifier-Free Guidance with $w = 4.0$.