Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector

TL;DR

This work tackles semantic drift in diffusion-model inference by injecting target representations through a lightweight representation alignment projector during intermediate denoising steps. The proposed R-Pred framework uses a learned representation predictor to form an implicit target, guiding the latent trajectory via gradient updates without modifying model architecture. Across SiT and REPA backbones on ImageNet 256256, representation-guided sampling yields consistent FID improvements and synergizes with classifier-free guidance, achieving substantial gains such as lowering FID from 5.9 to 3.3 in REPA-XL/2. The results demonstrate that training-free, representation-informed diffusion sampling can reinforce semantic preservation and image consistency in large-scale diffusion transformers with practical computational overhead.

Abstract

Recent progress in generative modeling has enabled high-quality visual synthesis with diffusion-based frameworks, supporting controllable sampling and large-scale training. Inference-time guidance methods such as classifier-free and representative guidance enhance semantic alignment by modifying sampling dynamics; however, they do not fully exploit unsupervised feature representations. Although such visual representations contain rich semantic structure, their integration during generation is constrained by the absence of ground-truth reference images at inference. This work reveals semantic drift in the early denoising stages of diffusion transformers, where stochasticity results in inconsistent alignment even under identical conditioning. To mitigate this issue, we introduce a guidance scheme using a representation alignment projector that injects representations predicted by a projector into intermediate sampling steps, providing an effective semantic anchor without modifying the model architecture. Experiments on SiTs and REPAs show notable improvements in class-conditional ImageNet synthesis, achieving substantially lower FID scores; for example, REPA-XL/2 improves from 5.9 to 3.3, and the proposed method outperforms representative guidance when applied to SiT models. The approach further yields complementary gains when combined with classifier-free guidance, demonstrating enhanced semantic coherence and visual fidelity. These results establish representation-informed diffusion sampling as a practical strategy for reinforcing semantic preservation and image consistency.

Figures (9)

• Figure 1: Class-conditional generation on ImageNet 256×256 without CFG using ODE sampling. “+G” denotes that the corresponding SiT/REPA model uses guidance. Applying Representation Guidance yields significantly better generation quality than state-of-the-art diffusion/flow models.
• Figure 2: Overview of the Representation Guidance method. Left: A representation projector from a pretrained model (e.g., REPA) is used as an indicator to evaluate and guide the current denoising result via correlation. Right: Guidance is applied only within an intermediate timestep range to enhance semantic fidelity, without disrupting the initial coarse sampling or the final refinement stage.
• Figure 3: Observation of representation similarity of generated results (with a reference image). We randomly selected an image from class 0 of ImageNet and applied varying noise levels: $x_t = (1-t)x_0 + t \cdot \text{noise}$. (a) We observed the denoising process for different seeds and note that each target image has an associated noise pattern that best aligns with it, which can be considered as the "golden" noise for that target. (b) We examined the denoising outputs of pretrained generative models (SiT-XL/2, REPA-XL/2) and the representations directly predicted by REPA's projector, comparing their similarity to the reference image. (c) Using the projector-predicted representations to guide the generative model.
• Figure 4: Selected samples on ImageNet 256$\times$256 from the REPA-XL/2 model (top) and with guidance (bottom). Applying Representation Guidance results in noticeable improvements in the generated images. We did not use classifier-free guidance, so $w = 1.0$.
• Figure 5: ImageNet256x256/class: tiger shark. The images on the left, preceding the arrow, represent incorrect outputs produced by REPA-XL/2 in the absence of CFG and other guidance, whereas the images on the right, following the arrow, illustrate the improvements obtained through R-pred guidance..