Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation

TL;DR

This work addresses the trade-off in diffusion models between end-to-end pixel-space modeling and tokenized latent representations. It introduces Latent Forcing, a pixel-space diffusion approach that animates two time variables to denoise latents before pixels, effectively using latents as a scratchpad while preserving full information without tokenization losses. The authors provide extensive analysis and ablations on ordering, time scheduling, guidance, and distillation, showing state-of-the-art results for pixel-diffusion transformers on ImageNet at the same compute scale. The approach is end-to-end at inference with minimal architectural changes and demonstrates that generation quality can be driven by the ordering of conditioning signals rather than tokenization alone, enabling practical, scalable improvements in pixel-space diffusion.

Abstract

Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, conditional versus unconditional generation, and how tokenizer reconstruction quality relates to diffusability. Applied to ImageNet, Latent Forcing achieves a new state-of-the-art for diffusion transformer-based pixel generation at our compute scale.

Figures (6)

• Figure 1: Conceptual Diagram. We diffuse both latents and pixels together, each with their own time variable. This allows us to denoise an easier trajectory than pure pixel diffusion by diffusing latents first, leading to improved performance.
• Figure 2: Model Architecture. Our approach makes minimal changes to diffusion transformers. Left: We add the per-patch embeddings of latents and pixels together, keeping the same number of tokens. Middle: We use two time variables instead of one for adaLN. Right: Optionally, we take the last $M=4$ transformer layers and split them into two $M/2$-layer output experts.
• Figure 3: FID-10K values for different diffusion trajectories through a joint DINOv2 and Pixel Space.
• Figure 4: Output single-step $\mathbf{x}$-predictions in the pixel space from the Multi-Schedule Model, where each column reconstructs with the same PSNR. Top: Predictions when pixel features are partially denoised, and DINOv2 features are fully noised. Bottom: Predictions when latent DINOv2 features are partially denoised and pixels are fully noised. At low PSNR levels, DINOv2 features preserve significantly more spatial information.
• Figure 5: PSNRs on DINOv2 and Pixel features at different timestep combinations in the Multi-Schedule Model.