DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer

TL;DR

DreamID-V addresses the challenge of high-fidelity, temporally coherent video face swapping by bridging the gap between image-based and video-based methods. It introduces SyncID-Pipe to generate paired supervision via an Identity-Anchored Video Synthesizer and bidirectional ID quadruplets, and DreamID-V, a Diffusion Transformer with Modality-Aware Conditioning, trained through Synthetic-to-Real Curriculum and Identity-Coherence Reinforcement Learning. The authors also present IDBench-V to benchmark VFS across diverse scenes. Empirical results show DreamID-V outperforms state-of-the-art on identity similarity, attribute preservation, and video quality, and the framework is versatile enough to extend to other swap-related tasks.

Abstract

Video Face Swapping (VFS) requires seamlessly injecting a source identity into a target video while meticulously preserving the original pose, expression, lighting, background, and dynamic information. Existing methods struggle to maintain identity similarity and attribute preservation while preserving temporal consistency. To address the challenge, we propose a comprehensive framework to seamlessly transfer the superiority of Image Face Swapping (IFS) to the video domain. We first introduce a novel data pipeline SyncID-Pipe that pre-trains an Identity-Anchored Video Synthesizer and combines it with IFS models to construct bidirectional ID quadruplets for explicit supervision. Building upon paired data, we propose the first Diffusion Transformer-based framework DreamID-V, employing a core Modality-Aware Conditioning module to discriminatively inject multi-model conditions. Meanwhile, we propose a Synthetic-to-Real Curriculum mechanism and an Identity-Coherence Reinforcement Learning strategy to enhance visual realism and identity consistency under challenging scenarios. To address the issue of limited benchmarks, we introduce IDBench-V, a comprehensive benchmark encompassing diverse scenes. Extensive experiments demonstrate DreamID-V outperforms state-of-the-art methods and further exhibits exceptional versatility, which can be seamlessly adapted to various swap-related tasks.

Figures (13)

• Figure 1: Showcase of DreamID-V. DreamID-V robustly handles challenging scenarios, e.g., complex expressions, animation, large angles, occlusions, and small faces.
• Figure 2: Overview of SyncID-Pipe. We pre-train the Identity-Anchored Video Synthesizer and combine it with the Image Face Swapping model to construct Bidirectional Quadruplet Pair data.
• Figure 3: Overview of DreamID-V framework. We design customized injection mechanisms for Spatio-Temporal Context, Structural Guidance, and Identity Information, respectively.
• Figure 4: Qualitative comparisons with state-of-the-art methods. Please zoom in for more details.
• Figure 5: Ablation studies of DreamID-V.