DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation

TL;DR

DeCo introduces frequency-decoupled pixel diffusion, splitting high-frequency detail generation from low-frequency semantics by using a lightweight Pixel Decoder conditioned on a DiT-derived semantic cue. A JPEG-inspired frequency-aware Flow-Matching loss directs learning toward perceptually important frequencies, yielding clear gains in image fidelity and training/inference efficiency. Empirical results on ImageNet show leading pixel-diffusion FID scores ($1.62$ at $256\times256$, $2.22$ at $512\times512$) and strong GenEval performance ($0.86$), narrowing the gap with latent-diffusion methods. The approach demonstrates practical impact for high-quality, end-to-end image generation with reduced computational burden, and is complemented by extensive ablations validating architectural choices and loss design.

Abstract

Pixel diffusion aims to generate images directly in pixel space in an end-to-end fashion. This approach avoids the limitations of VAE in the two-stage latent diffusion, offering higher model capacity. Existing pixel diffusion models suffer from slow training and inference, as they usually model both high-frequency signals and low-frequency semantics within a single diffusion transformer (DiT). To pursue a more efficient pixel diffusion paradigm, we propose the frequency-DeCoupled pixel diffusion framework. With the intuition to decouple the generation of high and low frequency components, we leverage a lightweight pixel decoder to generate high-frequency details conditioned on semantic guidance from the DiT. This thus frees the DiT to specialize in modeling low-frequency semantics. In addition, we introduce a frequency-aware flow-matching loss that emphasizes visually salient frequencies while suppressing insignificant ones. Extensive experiments show that DeCo achieves superior performance among pixel diffusion models, attaining FID of 1.62 (256x256) and 2.22 (512x512) on ImageNet, closing the gap with latent diffusion methods. Furthermore, our pretrained text-to-image model achieves a leading overall score of 0.86 on GenEval in system-level comparison. Codes are publicly available at https://github.com/Zehong-Ma/DeCo.

Figures (10)

• Figure 1: Qualitative results of text-to-image generation of DeCo. All images are 512$\times$512 resolution.
• Figure 2: Illustration of our frequency-decoupled (DeCo) framework. In (a), traditional baseline models rely on a single DiT to jointly model both low-frequency semantics and high-frequency signals. (b) shows our DeCo framework, where a lightweight pixel decoder focuses on the high-frequency reconstruction, and the DiT models low-frequency semantics. As shown in (c), decoupling DiT from modeling high-frequency signals leads to better low-frequency semantic features in DiT Output, and higher image quality.
• Figure 3: Overview of the proposed frequency-decoupled (DeCo) framework. The DiT operates on downsampled inputs to model low-frequency semantics, while the lightweight pixel decoder generates high-frequency details under the DiT's semantic guidance.
• Figure 4: DCT energy distribution of DiT outputs and predicted pixel velocities. Compared with baseline, DeCo suppresses high-frequency signals in DiT outputs while preserving strong high-frequency energy in pixel velocity, confirming effective frequency decoupling. The distribution is computed on 10K images across all diffusion steps using DCT transform with 8$\times$8 block size.
• Figure 5: FID comparison between our DeCo and baseline. DeCo reaches 2.57 FID in 400k iterations, 10× faster than the baseline.