Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspective

TL;DR

The paper analyzes how high-frequency components in high-dimensional latent spaces affect reconstruction and generation in diffusion-based visual synthesis. By perturbing frequencies in RGB and latent domains, it finds that decoders rely on high-frequency latent content for details while encoders underrepresent extreme high frequencies, especially as channel count grows. It introduces FreqWarm, a plug-and-play frequency warm-up that filters RGB high-frequency signals to boost high-frequency latent energy early in training without retraining autoencoders, yielding consistent gFID gains across multiple high-dimensional tokenizers and diffusion backbones. The results demonstrate that managing frequency exposure improves diffusibility in high-dimensional latent spaces, enabling more aggressive compression while preserving generation quality and suggesting a co-design path for autoencoders and diffusion architectures with explicit frequency budgets.

Abstract

Latent diffusion has become the default paradigm for visual generation, yet we observe a persistent reconstruction-generation trade-off as latent dimensionality increases: higher-capacity autoencoders improve reconstruction fidelity but generation quality eventually declines. We trace this gap to the different behaviors in high-frequency encoding and decoding. Through controlled perturbations in both RGB and latent domains, we analyze encoder/decoder behaviors and find that decoders depend strongly on high-frequency latent components to recover details, whereas encoders under-represent high-frequency contents, yielding insufficient exposure and underfitting in high-frequency bands for diffusion model training. To address this issue, we introduce FreqWarm, a plug-and-play frequency warm-up curriculum that increases early-stage exposure to high-frequency latent signals during diffusion or flow-matching training -- without modifying or retraining the autoencoder. Applied across several high-dimensional autoencoders, FreqWarm consistently improves generation quality: decreasing gFID by 14.11 on Wan2.2-VAE, 6.13 on LTX-VAE, and 4.42 on DC-AE-f32, while remaining architecture-agnostic and compatible with diverse backbones. Our study shows that explicitly managing frequency exposure can successfully turn high-dimensional latent spaces into more diffusible targets.

Figures (7)

• Figure 1: Trade-off between reconstruction and generation. Reconstruction is evaluated by FID between input images and reconstructed images (i.e., rFID). Generation is evaluated by the FID between synthetic images and real images (i.e., gFID). Lower rFID and gFID indicate the better performance. The spatial compression ratio remains 32 for all experiments.
• Figure 2: Visualization of images reconstructed from low-frequency and high-frequency embeddings in the latent space. $r$ is the threshold to separate low frequency and high frequency. Please zoom in for more details.
• Figure 3: Visualization of images reconstructed from low-frequency and high-frequency components in the RGB space. $r$ is the threshold to separate low frequency and high frequency. Please zoom in for more details.
• Figure 4: Different frequency distributions in the latent space with regard to the low-pass threshold on RGB images (measured on 50k images). The spatial frequency of x-axis is measured by the distance to the center of frequency spectrum. Both axes are in logarithmic scale. Detailed explanation is described in \ref{['sec:analysis']}.
• Figure 5: Overview of FreqWarm. We filter out high-frequency components above a frequency threshold $r_0$ in the RGB space. The filtered images are forwarded into a pretrained autoencoder. We train diffusion models or flow matching models on top of the latent space in the early training stage for warm-up. Note that the autoencoder is kept frozen throughout training in our method.