Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion

TL;DR

Semantics Lead the Way introduces Semantic-First Diffusion (SFD), a latent diffusion framework that explicitly prioritizes semantic formation by denoising semantic latents earlier than texture latents in an asynchronous, three-phase schedule. It builds a composite latent by fusing SemVAE-derived semantic latents with SD-VAE texture latents and leverages a diffusion transformer with dual timesteps to guide texture refinement. The approach achieves state-of-the-art FID on ImageNet 256×256 with guidance (as low as 1.04) and dramatically accelerates convergence (up to ~100× faster than strong baselines), while generalizing to other semantic-texture models like ReDi and VA-VAE and preserving reconstruction fidelity via SD-VAE. These results suggest that representation-level asynchronous denoising, with semantics leading textures, can significantly enhance diffusion-based image synthesis and offers directions for broader multimodal generation tasks.

Abstract

Latent Diffusion Models (LDMs) inherently follow a coarse-to-fine generation process, where high-level semantic structure is generated slightly earlier than fine-grained texture. This indicates the preceding semantics potentially benefit texture generation by providing a semantic anchor. Recent advances have integrated semantic priors from pretrained visual encoders to further enhance LDMs, yet they still denoise semantic and VAE-encoded texture synchronously, neglecting such ordering. Observing these, we propose Semantic-First Diffusion (SFD), a latent diffusion paradigm that explicitly prioritizes semantic formation. SFD first constructs composite latents by combining a compact semantic latent, which is extracted from a pretrained visual encoder via a dedicated Semantic VAE, with the texture latent. The core of SFD is to denoise the semantic and texture latents asynchronously using separate noise schedules: semantics precede textures by a temporal offset, providing clearer high-level guidance for texture refinement and enabling natural coarse-to-fine generation. On ImageNet 256x256 with guidance, SFD achieves FID 1.06 (LightningDiT-XL) and FID 1.04 (1.0B LightningDiT-XXL), while achieving up to 100x faster convergence than the original DiT. SFD also improves existing methods like ReDi and VA-VAE, demonstrating the effectiveness of asynchronous, semantics-led modeling. Project page and code: https://yuemingpan.github.io/SFD.github.io/.

Figures (17)

• Figure 1: (a) Overview of Semantic-First Diffusion (SFD). Semantics (dashed curve) and textures (solid curve) follow asynchronous denoising trajectories. SFD operates in three phases: Stage $\mathrm{I}$ -- Semantic initialization, where semantic latents denoise first; Stage $\mathrm{II}$ -- Asynchronous generation, where semantics and textures denoise jointly but asynchronously, with semantics ahead of textures; Stage $\mathrm{III}$ -- Texture completion, where only textures continue refining. After denoising, the generated semantic latent $\mathbf{s}_1$ is discarded, and the final image is decoded solely from the texture latent $\mathbf{z}_1$. (b) Training convergence on ImageNet 256$\times$256 without guidance. SFD achieves substantially faster convergence than DiT-XL/2 and LightningDiT-XL/1 by approximately 100$\times$ and 33.3$\times$, respectively.
• Figure 2: Architecture of the Semantic VAE (SemVAE). A Transformer-based VAE compresses high-dimensional vision foundation model (VFM) features into compact semantic latents.
• Figure 3: Composite Latent Construction. An input image is encoded into semantic and texture latents via distinct VAE encoders, which are then concatenated to form a composite latent for asynchronous diffusion modeling.
• Figure 4: Input and output of Diffusion Transformer. A DiT backbone takes as input a composite latent that combines noisy semantic and texture features $[\mathbf{s}_{t_s}, \mathbf{z}_{t_z}]$, along with their respective timestep $[t_s, t_z]$ and class label $y$. It jointly predicts the velocities of both semantics and textures.
• Figure 5: Effect of the temporal offset $\Delta t$ in asynchronous denoising. A moderate offset ($\Delta t=$ 0.3) yields the lowest FID, indicating the best semantic--texture cooperation.