Decoupling Complexity from Scale in Latent Diffusion Model

TL;DR

This work tackles the problem of latent diffusion models tying information complexity to data scale. It introduces DCS-LDM, a framework with a scale-independent latent space and hierarchical multi-level tokens, enabling decoding at arbitrary resolutions and frame rates from a fixed latent capacity. Through level-caused, coarse-to-fine generation and a dedicated tokenizer (DCS-Tok) and diffusion model (DCS-DiT), it achieves flexible compute-quality tradeoffs while maintaining competitive reconstruction and generation performance. The approach further employs content-adaptive token allocation and temporal-spatial causality, showing strong results on images and videos and offering rapid previews and progressive refinement.

Abstract

Existing latent diffusion models typically couple scale with content complexity, using more latent tokens to represent higher-resolution images or higher-frame rate videos. However, the latent capacity required to represent visual data primarily depends on content complexity, with scale serving only as an upper bound. Motivated by this observation, we propose DCS-LDM, a novel paradigm for visual generation that decouples information complexity from scale. DCS-LDM constructs a hierarchical, scale-independent latent space that models sample complexity through multi-level tokens and supports decoding to arbitrary resolutions and frame rates within a fixed latent representation. This latent space enables DCS-LDM to achieve a flexible computation-quality tradeoff. Furthermore, by decomposing structural and detailed information across levels, DCS-LDM supports a progressive coarse-to-fine generation paradigm. Experimental results show that DCS-LDM delivers performance comparable to state-of-the-art methods while offering flexible generation across diverse scales and visual qualities.

Figures (15)

• Figure 1: Overview of DCS-LDM.
• Figure 2: DCS-Tok adopts a pure Transformer architecture, comprising a 1D patchifier–depatchifier and a causal encoder–decoder. The patchifier–depatchifier align scale-aware spatial–temporal patches into a causal token sequence, where lower-level tokens capture structure and higher-level tokens encode fine-grained textures. The encoder–decoder facilitate cross-patch interaction while preserving causal dependencies across token levels.
• Figure 3: Detailed architectures. (a) Patchifier and depatchifier of DCS-Tok. (b) Encoder and decoder of DCS-Tok. (c) DCS-DiT. PLA, LPA, and LCA denote Pixel-to-Latent, Latent-to-Pixel, and Level Causal Attention, respectively. $L_i$ indicates the number of blocks.
• Figure 4: Coarse-to-fine generation. Causality enables computations to be separated by token levels. Caching keys and values from previous levels keeps the computational cost of progressive generation identical to that of non-progressive generation.
• Figure 5: Image reconstruction via DCS-Tok across samples with varying information complexity, resolution, and latent token levels. Brighter background indicates higher PSNR. Configurations with comparable PSNR are highlighted in bold.