DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning

TL;DR

DreamID tackles the lack of explicit supervision in diffusion-based face swapping by introducing Triplet ID Group Learning, which constructs triplets ($A_1$, $\tilde{B}$, $A_2$) using a GAN proxy to provide near-ground-truth supervision. Paired with SD Turbo single-step diffusion, this approach enables end-to-end training with image-space losses, while an architecture of SwapNet, FaceNet, and ID Adapter fuses pixel-level and semantic-level identity cues. The method achieves state-of-the-art results on FFHQ-style data, delivering high identity similarity and attribute preservation with 1-step inference in about $0.6$ seconds at $512\times512$, and demonstrates robustness to challenging lighting, angles, and occlusions. This combination of explicit supervision and fast diffusion inference advances practical, high-fidelity face swapping with fine-grained attribute control.

Abstract

In this paper, we introduce DreamID, a diffusion-based face swapping model that achieves high levels of ID similarity, attribute preservation, image fidelity, and fast inference speed. Unlike the typical face swapping training process, which often relies on implicit supervision and struggles to achieve satisfactory results. DreamID establishes explicit supervision for face swapping by constructing Triplet ID Group data, significantly enhancing identity similarity and attribute preservation. The iterative nature of diffusion models poses challenges for utilizing efficient image-space loss functions, as performing time-consuming multi-step sampling to obtain the generated image during training is impractical. To address this issue, we leverage the accelerated diffusion model SD Turbo, reducing the inference steps to a single iteration, enabling efficient pixel-level end-to-end training with explicit Triplet ID Group supervision. Additionally, we propose an improved diffusion-based model architecture comprising SwapNet, FaceNet, and ID Adapter. This robust architecture fully unlocks the power of the Triplet ID Group explicit supervision. Finally, to further extend our method, we explicitly modify the Triplet ID Group data during training to fine-tune and preserve specific attributes, such as glasses and face shape. Extensive experiments demonstrate that DreamID outperforms state-of-the-art methods in terms of identity similarity, pose and expression preservation, and image fidelity. Overall, DreamID achieves high-quality face swapping results at 512*512 resolution in just 0.6 seconds and performs exceptionally well in challenging scenarios such as complex lighting, large angles, and occlusions.

Figures (19)

• Figure 1: DreamID can generate high fidelity face swapping results at 512 × 512 resolution. In each group, we present the swapped face on the right, which is created by replacing the source face (top-left) with the target face (bottom-left). Our model is capable of generating high-similarity face swaps and performs exceptionally well in a variety of challenging scenarios, including makeup preservation, large angles, stylization, and complex lighting conditions.
• Figure 2: (a) The typical face-swapping training process, which often relies on implicit supervision. (b) Unlike previous work, DreamID constructs Triplet ID Group data for explicit supervision.
• Figure 3: Overview of DreamID. (a)Triplet ID Group Training. We establish explicit supervision for face swapping by constructing Triplet ID Group data. The construction process utilizes two images with the same $\mathrm{ID}(A_1 ,A_2)$ and one image with a different $\mathrm{ID}_{B}$, along with a FaceSwap Proxy model, to generate a Pseudo Target $\mathrm{ID}_{\Tilde{B}}$. Additionally, we initialize our DreamID with SD Turbo, reducing the inference steps to a single step. This allows for convenient computation of image-space losses, such as ID Loss and reconstruction Loss. (b) DreamID Model architecture. Our model architecture is composed of three components:1) The base Unet, which we refer to as SwapNet, is responsible for the main process of face swapping. 2) the face Unet feature encoder, named FaceNet, which extracts pixel-level ID information of the user image. 3) the ID Adapter that extracts the semantic-level ID information of the user image. The core feature fusion computation process is illustrated at the bottom.
• Figure 4: Data construction for specific feature control.
• Figure 5: Qualitative comparison of state-of-the-art methods on the FFHQ dataset. DreamID demonstrates significant advantages in terms of similarity, natural blending, occlusion handling, and attribute preservation such as expression, lighting, and makeup.