Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers

TL;DR

Diffusion Transformers offer high-quality image generation but suffer from high latency and memory usage. DiffCR introduces token-level routing with a mixture-of-depths-inspired router and differentiable compression ratios that are learned across tokens, layers, and timesteps, enabling adaptive computation. It deploys three enablers: (i) token-level routing with joint fine-tuning, (ii) layer-wise differentiable ratios initialized at zero with discrete-bin proxies, and (iii) timestep-wise differentiable ratios with regional timesteps. Empirical results on image inpainting and text-to-image tasks show improved FID/CLIP metrics while maintaining or reducing latency and memory, surpassing strong baselines. This dynamic, end-to-end learnable framework paves the way for efficient deployment of diffusion transformers on resource-constrained devices.

Abstract

Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) image generation quality but suffer from high latency and memory inefficiency, making them difficult to deploy on resource-constrained devices. One major efficiency bottleneck is that existing DiTs apply equal computation across all regions of an image. However, not all image tokens are equally important, and certain localized areas require more computation, such as objects. To address this, we propose DiffCR, a dynamic DiT inference framework with differentiable compression ratios, which automatically learns to dynamically route computation across layers and timesteps for each image token, resulting in efficient DiTs. Specifically, DiffCR integrates three features: (1) A token-level routing scheme where each DiT layer includes a router that is fine-tuned jointly with model weights to predict token importance scores. In this way, unimportant tokens bypass the entire layer's computation; (2) A layer-wise differentiable ratio mechanism where different DiT layers automatically learn varying compression ratios from a zero initialization, resulting in large compression ratios in redundant layers while others remain less compressed or even uncompressed; (3) A timestep-wise differentiable ratio mechanism where each denoising timestep learns its own compression ratio. The resulting pattern shows higher ratios for noisier timesteps and lower ratios as the image becomes clearer. Extensive experiments on text-to-image and inpainting tasks show that DiffCR effectively captures dynamism across token, layer, and timestep axes, achieving superior trade-offs between generation quality and efficiency compared to prior works. The project website is available at https://www.haoranyou.com/diffcr.

Figures (13)

• Figure 1: Overview of the proposed DiffCR framework: (a) token-level routing scheme and (b) differentiable compression ratios.
• Figure 2: Visualization of the router's predictions: (a) For inpainting tasks, where inputs are masked images with text prompts, we follow the previous SOTA method Lazy-Diffusion nitzan2024lazy to generate only the masked area rather than the entire image; (b) For text-to-image (T2I) tasks, where inputs are noise and text prompts, we follow PixArt-$\Sigma$chen2024pixart for generation. Each visualization includes the router's prediction map with values ranging from 0 to 1. The generated image at each corresponding timestep is shown on the left, while the router's prediction maps across various layers and timesteps are displayed on the right. More visualizations are provided in the supplementary materials.
• Figure 3: Comparison of latency and memory savings between our DiffCR router and the previous token merging method (ToMe) bolya2023token when applied to ViT-XL/2 vitnitzan2024lazy on an A100 GPU.
• Figure 4: Visualization of the compression ratio trajectory during fine-tuning: (a) Trajectories for each of the 28 layers in DiT models; (b) Average ratio trajectory across all layers; and (c) The final learned ratio distribution across 28 layers.
• Figure 5: Visualization of the learned ratio patterns across both timesteps and layers for the (a) inpainting task and (b) T2I task.