RefAlign: Representation Alignment for Reference-to-Video Generation

Abstract

Reference-to-video (R2V) generation is a controllable video synthesis paradigm that constrains the generation process using both text prompts and reference images, enabling applications such as personalized advertising and virtual try-on. In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image and jointly feed them into the diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy--paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoder features. In this paper, we propose RefAlign, a representation alignment framework that explicitly aligns DiT reference-branch features to the semantic space of a visual foundation model (VFM). The core of RefAlign is a reference alignment loss that pulls the reference features and VFM features of the same subject closer to improve identity consistency, while pushing apart the corresponding features of different subjects to enhance semantic discriminability. This simple yet effective strategy is applied only during training, incurring no inference-time overhead, and achieves a better balance between text controllability and reference fidelity. Extensive experiments on the OpenS2V-Eval benchmark demonstrate that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference alignment for R2V tasks.

Figures (11)

• Figure 1: Reference-to-video generation using our proposed method, RefAlign.
• Figure 2: Motivation of the proposed RefAlign method. (a) The R2V task suffers from copy--paste artifacts (top) and multi-subject confusion (bottom), both generated by Kling Kling. (b) t-SNE JMLR:v9:vandermaaten08a visualization of reference feature distributions. DiT features (conditioned on VAE-encoded inputs) are highly entangled and overlap substantially across references, whereas DINOv3 features are more separable. RefAlign aligns DiT features to the DINOv3 feature space via an alignment loss, improving reference separability by pulling same-reference features closer and pushing different-reference features farther apart. (c) Visual comparison with and without RefAlign.
• Figure 3: (a) Overview of RefAlign. During training, we apply the proposed reference alignment loss $\mathcal{L}_{\mathrm{RA}}$ to intermediate features in selected DiT blocks and align them to target features extracted by a frozen vision foundation model (VFM). During inference, we discard the alignment process and the VFM. (b) Illustration of the reference alignment (RA) loss. RA loss aligns DiT reference features to their corresponding VFM teacher features by pulling matched (same-subject) pairs together and pushing mismatched (cross-subject) pairs apart, improving reference-consistent generation.
• Figure 4: Training comparison between REPA and RefAlign. (a) REPA: Trained from scratch, aligning noisy generation targets with clean VFM features to accelerate DiT convergence. (b) RefAlign: Fine-tuned from Wan2.1wan2025wan initialization, aligning clean reference-branch image features with clean VFM features to optimize reference representations and improve reference controllability.
• Figure 5: Qualitative results. We compare RefAlign with three representative methods, namely Kling1.6 Kling, Phantom Liu_2025_ICCV, and VINO chen2026vino.