Beyond External Guidance: Unleashing the Semantic Richness Inside Diffusion Transformers for Improved Training

TL;DR

The paper tackles slow convergence in diffusion transformers by removing reliance on external semantic supervision. It introduces Self-Transcendence, a two-stage self-guided framework that first aligns shallow features with stable VAE latents via an auxiliary $L_2$ loss and then enriches intermediate representations using classifier-free guidance in the deep feature space, combined with a self-guided loss $\mathcal{L}_{guide}$. The training objective becomes $\mathcal{L} = \mathcal{L}_{diff} + \lambda_{guide}\mathcal{L}_{guide}$ with a guiding equation $\mathbf{f}_{g} = \mathbf{f}_{u} + \omega \cdot (\mathbf{f}_{c} - \mathbf{f}_{u})$, enabling self-supervised semantic transfer. Experiments on SiT and LightningDiT backbones show competitive or superior generation quality (FID, IS) and faster convergence compared to externally guided methods like REPA, while avoiding pretrained vision encoders and extra data; the approach also demonstrates scalability and generality across backbones and resolutions, with an efficient warm-up overhead.

Abstract

Recent works such as REPA have shown that guiding diffusion models with external semantic features (e.g., DINO) can significantly accelerate the training of diffusion transformers (DiTs). However, this requires the use of pretrained external networks, introducing additional dependencies and reducing flexibility. In this work, we argue that DiTs actually have the power to guide the training of themselves, and propose \textbf{Self-Transcendence}, a simple yet effective method that achieves fast convergence using internal feature supervision only. It is found that the slow convergence in DiT training primarily stems from the difficulty of representation learning in shallow layers. To address this, we initially train the DiT model by aligning its shallow features with the latent representations from the pretrained VAE for a short phase (e.g., 40 epochs), then apply classifier-free guidance to the intermediate features, enhancing their discriminative capability and semantic expressiveness. These enriched internal features, learned entirely within the model, are used as supervision signals to guide a new DiT training. Compared to existing self-contained methods, our approach brings a significant performance boost. It can even surpass REPA in terms of generation quality and convergence speed, but without the need for any external pretrained models. Our method is not only more flexible for different backbones but also has the potential to be adopted for a wider range of diffusion-based generative tasks. The source code of our method can be found at https://github.com/csslc/Self-Transcendence.

Figures (10)

• Figure 1: Left: PCA visualization abdi2010pca of latent features from both shallow (layer 8) and deeper (layer 16) blocks of SiT with $t=0.6$ during training. Both layers progressively learn clean and discriminative representations, but the shallow layer learns such representations at a slower pace compared to the deeper one. Right: Comparison of guiding features from different methods. Our proposed approach produces clearer structural organization and more semantically richer features, as pre-trained DINO dinov2 used in REPA repa.
• Figure 2: t-SNE visualizations of the guiding features extracted from (a) REPA repa, (b) LayerSync LayerSync, (c) VAE features, and (d) our Self-Transcendence with $t=0.4$ in the 200K iteration of SiT-XL/2. Different colors represent different classes. As REPA, our internal guiding features demonstrate superior class separability.
• Figure 3: The framework of our proposed Self-Transcendence. The spark icon indicates that the parameters of this layer are trainable, while the snowflake icon indicates that they are frozen.
• Figure 4: Comparison of FID-10K scores across training iterations on ImageNet (256×256). VAE-based alignment accelerates SiT training, while leveraging this model for self-transcendence leads to further improvements.
• Figure 5: Visual comparison of generated samples from SiT-XL/2 models at different training iterations. For all models, we apply the same seed, noise, and sampling strategy with a CFG scale of 4.0.