Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation

TL;DR

Stand-In tackles identity-preserving video generation with a lightweight, plug-and-play approach. It injects a conditional image branch into a pre-trained diffusion video model and uses restricted self-attention with conditional position mapping to fuse identity cues while preserving the model's priors, training roughly 1% of additional parameters on about 2000 pairs. The method achieves state-of-the-art identity fidelity and naturalness, with strong prompt-following and minimal runtime overhead, and it proves versatile across tasks like pose-guided generation, stylization, and face swapping. This demonstrates a practical pathway to high-quality, identity-consistent videos that can readily integrate with existing AIGC pipelines and non-human subjects.

Abstract

Generating high-fidelity human videos that match user-specified identities is important yet challenging in the field of generative AI. Existing methods often rely on an excessive number of training parameters and lack compatibility with other AIGC tools. In this paper, we propose Stand-In, a lightweight and plug-and-play framework for identity preservation in video generation. Specifically, we introduce a conditional image branch into the pre-trained video generation model. Identity control is achieved through restricted self-attentions with conditional position mapping. Thanks to these designs, which greatly preserve the pre-trained prior of the video generation model, our approach is able to outperform other full-parameter training methods in video quality and identity preservation, even with just $\sim$1% additional parameters and only 2000 training pairs. Moreover, our framework can be seamlessly integrated for other tasks, such as subject-driven video generation, pose-referenced video generation, stylization, and face swapping.

Figures (21)

• Figure 1: Given a reference image, our method generates videos with strong identity preservation. Furthermore, the framework's plug-and-play design enables seamless integration into diverse applications for enhanced identity consistency.
• Figure 2: Comparison with SOTA identity-preserving video generation methods. The size of bubbles represents the number of need-to-train parameters for identity preservation. Our approach achieves the highest performance in both face similarity and naturalness, while utilizing the fewest parameters.
• Figure 3: The overview of our identity-preserving text-to-video generation framework. We introduce a conditional image branch alongside the original video branch. Given the conditional image, the VAE encoder maps it into tokens, which are concatenated with the video latent tokens and then sent to the DiT. Within the DiT blocks, identity information is incorporated into the video features through restricted self-attention.
• Figure 4: Design of our Restricted Self-Attention: For the input video and image tokens, we compute their Query, Key, and Value matrices independently. Next, we apply Conditional Position Mapping to the Query and Key matrices. Finally, the image matrices operate independently, while the video Query performs attention using the concatenation of the image and video Key and Value matrices.
• Figure 5: Effect of Restricted Self-Attention (RSA). Given the reference image and the prompt (left column), we visualize attention map to reference-image tokens. Under Vanilla Self-Attention (top row), attention diffuses into background regions and the output skews toward a garden scene. With RSA (bottom row), attention concentrates on facial regions, maintaining the subject’s identity.