DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space

TL;DR

Addressing slow diffusion-based image synthesis at high resolutions, the paper introduces DC-Gen, a post-training framework that couples pretrained diffusion models with deeply compressed autoencoders. It stabilizes adaptation through an embedding-alignment stage before lightweight LoRA fine-tuning, preserving the base model's knowledge while dramatically reducing token counts. Empirical results on SANA and FLUX demonstrate comparable quality to the base models but with substantial speedups, including ~53x latency reduction at 4K and up to ~138x total latency reduction when combined with quantization. This approach enables native high-resolution generation with markedly lower training and inference costs, broadening practical deployment of high-fidelity diffusion models.

Abstract

Existing text-to-image diffusion models excel at generating high-quality images, but face significant efficiency challenges when scaled to high resolutions, like 4K image generation. While previous research accelerates diffusion models in various aspects, it seldom handles the inherent redundancy within the latent space. To bridge this gap, this paper introduces DC-Gen, a general framework that accelerates text-to-image diffusion models by leveraging a deeply compressed latent space. Rather than a costly training-from-scratch approach, DC-Gen uses an efficient post-training pipeline to preserve the quality of the base model. A key challenge in this paradigm is the representation gap between the base model's latent space and a deeply compressed latent space, which can lead to instability during direct fine-tuning. To overcome this, DC-Gen first bridges the representation gap with a lightweight embedding alignment training. Once the latent embeddings are aligned, only a small amount of LoRA fine-tuning is needed to unlock the base model's inherent generation quality. We verify DC-Gen's effectiveness on SANA and FLUX.1-Krea. The resulting DC-Gen-SANA and DC-Gen-FLUX models achieve quality comparable to their base models but with a significant speedup. Specifically, DC-Gen-FLUX reduces the latency of 4K image generation by 53x on the NVIDIA H100 GPU. When combined with NVFP4 SVDQuant, DC-Gen-FLUX generates a 4K image in just 3.5 seconds on a single NVIDIA 5090 GPU, achieving a total latency reduction of 138x compared to the base FLUX.1-Krea model. Code: https://github.com/dc-ai-projects/DC-Gen.

Figures (13)

• Figure 1: High-Resolution Image Samples Generated by DC-Gen-FLUX. DC-Gen-FLUX enables native 4K image generation with about 53$\times$ latency reduction over the base model FLUX.1-Krea, measured on an NVIDIA H100 GPU.
• Figure 2: Illustration of DC-Gen. (a) DC-Gen accelerates diffusion models by integrating pretrained diffusion models with deeply compressed autoencoders (e.g., DC-AE chen2024deep). (b) DC-Gen introduces an embedding alignment stage to mitigate discrepancies between embedding spaces, thereby preventing training instability and preserve base model's knowledge. (c) DC-Gen achieves drastic reductions in training cost relative to training from scratch. DC-Gen-FLUX-12B requires only 40 H100 GPU days, representing a 520$\times$ reduction compared to DALL·E 2-6.5B.
• Figure 3: Challenge in Shifting Latent Space for a Pretrained Model. When replacing the pretrained DiT's latent space from SD-VAE-f8 to DC-AE-f32, standard fine-tuning fails to reach the base model's performance and is unstable. In contrast, training with DC-Gen is stable and even slightly improves upon the base model's FID. Detailed results are provided in Table. \ref{['tab:ablation_embeding_alignment']} and Fig. \ref{['fig:appendix_full_ablation']}.
• Figure 4: (a) Embedding alignment significantly reduces the per-layer representation gap with the pretrained diffusion model. (b) After embedding alignment, the model can generate images with correct semantics in the new latent space, even without fine-tuning the diffusion model's weights.
• Figure 5: Ablation of Tuning Strategies in End-to-End Fine-Tuning. LoRA tuning better preserves the base model's knowledge during end-to-end fine-tuning and achieves higher quality. Metrics are reported on 1K samples from MJHQ-30K at 512$\times$512 resolution.