Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints

TL;DR

This work identifies Dynamic Feature Instability as a core challenge in Diffusion Transformers (DiT) and shows that unstable feature propagation stems from uncontrolled spectral norms in DiT. It introduces Skip-DiT, a Long-Skip-Connections (LSCs)–driven DiT with spectral-constrained weights, providing theoretical guarantees that the Jacobian spectral norm is tighter than in vanilla DiT, enabling stable gradients. The authors couple this architectural change with a static, cache-friendly inference strategy that reuses deep features across timesteps, achieving up to 4.4× training acceleration and 1.5–2× inference speedups with negligible quality loss across image and video generation tasks. Extensive experiments across multiple backbones (Latte, Hunyuan-DiT, DiT-XL) and datasets demonstrate improved stability and caching efficiency, with compatibility to other caching methods, offering a practical path toward stabilized and efficient diffusion transformers for vision applications.

Abstract

Diffusion Transformers (DiT) have emerged as a powerful architecture for image and video generation, offering superior quality and scalability. However, their practical application suffers from inherent dynamic feature instability, leading to error amplification during cached inference. Through systematic analysis, we identify the absence of long-range feature preservation mechanisms as the root cause of unstable feature propagation and perturbation sensitivity. To this end, we propose Skip-DiT, an image and video generative DiT variant enhanced with Long-Skip-Connections (LSCs) - the key efficiency component in U-Nets. Theoretical spectral norm and visualization analysis demonstrate how LSCs stabilize feature dynamics. Skip-DiT architecture and its stabilized dynamic feature enable an efficient statical caching mechanism that reuses deep features across timesteps while updating shallow components. Extensive experiments across the image and video generation tasks demonstrate that Skip-DiT achieves: (1) 4.4 times training acceleration and faster convergence, (2) 1.5-2 times inference acceleration with negligible quality loss and high fidelity to the original output, outperforming existing DiT caching methods across various quantitative metrics. Our findings establish Long-Skip-Connections as critical architectural components for stable and efficient diffusion transformers. Codes are provided in the https://github.com/OpenSparseLLMs/Skip-DiT.

Figures (8)

• Figure 1: Feature stability comparison between Skip-DiT (left) and vanilla DiT (right) on class-to-image (DiT-XL) and text-to-video (Latte) generation. We inject perturbations $\delta$ (magnitudes $\alpha$) and $\eta$ (magnitudes $\beta$), normalized with specific coefficients ($\epsilon = 1e^{-3}$ for DiT-XL and $2e^{-2}$ for Latte), then measure the similarity between the standard and perturbed features.
• Figure 2: Comparison between standard and cache-accelerated outputs in vanilla DiT versus Skip-DiT, with Latte (text-to-video generation) and DiT-XL (class-to-image generation) serving as base architectures. Both mean and standard deviation across samples are shown. Vanilla DiT exhibits much higher sample variance.
• Figure 3: Training efficiency comparison between Skip-DiT and DiT-XL. Skip-DiT achieves superior FID-50K score on ImageNet at 1.6M training steps (vs. DiT-XL’s 7M steps) and converges faster at 2.9M steps. Both models are evaluated on FID-50K under identical settings (no classifier-free guidance, cfg=1).
• Figure 4: The feature dynamics of Latte with LSCs. Differences in features at the same layers across timesteps are evaluated.
• Figure 5: Qualitative results of text-to-video generation. We present Skip-DiT , PAB$_{469}$, and the original model. The frames are randomly sampled from the generated video.

Theorems & Definitions (4)

• Theorem 1
• Theorem 2
• proof
• proof