EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusion

TL;DR

EchoVideo addresses the challenge of identity-preserving video generation by mitigating copy-paste artifacts and semantic conflicts through an Identity Image-Text Fusion Module (IITF) that harmonizes high-level text semantics with facial identity features. It introduces a two-stage training regime that uses shallow facial information stochastically to balance fidelity and generalization, enabling robust identity preservation and full-body consistency. The method leverages a DiT-based video generator with pre-fusion multimodal guidance and is trained on a curated multi-type dataset to diversify pose and lighting while reducing over-reliance on input faces. Experiments demonstrate state-of-the-art identity preservation and competitive video quality, with a plug-and-play design that can extend to other pre-trained diffusion or U-Net-based models.

Abstract

Recent advancements in video generation have significantly impacted various downstream applications, particularly in identity-preserving video generation (IPT2V). However, existing methods struggle with "copy-paste" artifacts and low similarity issues, primarily due to their reliance on low-level facial image information. This dependence can result in rigid facial appearances and artifacts reflecting irrelevant details. To address these challenges, we propose EchoVideo, which employs two key strategies: (1) an Identity Image-Text Fusion Module (IITF) that integrates high-level semantic features from text, capturing clean facial identity representations while discarding occlusions, poses, and lighting variations to avoid the introduction of artifacts; (2) a two-stage training strategy, incorporating a stochastic method in the second phase to randomly utilize shallow facial information. The objective is to balance the enhancements in fidelity provided by shallow features while mitigating excessive reliance on them. This strategy encourages the model to utilize high-level features during training, ultimately fostering a more robust representation of facial identities. EchoVideo effectively preserves facial identities and maintains full-body integrity. Extensive experiments demonstrate that it achieves excellent results in generating high-quality, controllability and fidelity videos.

Figures (7)

• Figure 1: Sampling results of EchoVideo. (a) Facial feature preservation. (b) Full-body feature preservation. EchoVideo is capable of not only extracting human features but also resolving semantic conflicts between these features and the prompt, thereby generating coherent and consistent videos.
• Figure 2: Issues in IP character generation. (a) Semantic conflict. The input image depicts a child's face, while the prompt specifies an adult male. Insufficient information interaction leads to inconsistent character traits in the model's output. (b) Copy-paste. During training, the model overly relies on visual information from facial images, directly using the Variational Autoencoder(VAE)-encoded yang2020causalvae face as the output for the generated face.
• Figure 3: Overall architecture of EchoVideo. By employing a meticulously designed IITF module and mitigating the over-reliance on input images, our model effectively unifies the semantic information between the input facial image and the textual prompt. This integration enables the generation of consistent characters with multi-view facial coherence, ensuring that the synthesized outputs maintain both visual and semantic fidelity across diverse perspectives.
• Figure 4: Illustration of facial information injection methods. (a) Dual branch. Facial and textual information are independently injected through Cross Attention mechanisms, providing separate guidance for the generation process. (b) IITF. Facial and textual information are fused to ensure consistent guidance throughout the generation process.
• Figure 5: Qualitative results. (a) Ours. (b) ConsisID yuan2024identity. (c) ID-Animator he2024id. Our model can effectively overcome semantic conflicts and copy-paste phenomena while maintaining the face IP.