GaussianSwap: Animatable Video Face Swapping with 3D Gaussian Splatting

TL;DR

GaussianSwap tackles the limitation of pixel-based video face swapping by constructing an animatable 3D Gaussian Splatting (3DGS) head avatar from a target video and transferring identity from a source image. The approach tightly integrates FLAME-based tracking, per-frame Gaussian bindings to a FLAME mesh, and identity finetuning via a compound embedding from ArcFace, FaceNet, and Dlib to produce high-fidelity, temporally coherent face-swapped avatars and videos. Key contributions include (1) a 3DGS-based avatar construction pipeline conditioned on full-head FLAME data, (2) a compound identity loss for robust identity transfer, and (3) a rendering and fusion framework that enables downstream interactive applications such as video reenactment, speech-driven avatar animation, and dynamic background manipulation. This work enables new practical scenarios where face-swapped content remains animatable and integrable into virtual environments, while maintaining competitive visual quality and temporal stability. Limitations arise from FLAME tracking limits in capturing fine expressions and from potential side-view artifacts when target views are limited, pointing to future work on improved 3D head tracking and view-generalization.

Abstract

We introduce GaussianSwap, a novel video face swapping framework that constructs a 3D Gaussian Splatting based face avatar from a target video while transferring identity from a source image to the avatar. Conventional video swapping frameworks are limited to generating facial representations in pixel-based formats. The resulting swapped faces exist merely as a set of unstructured pixels without any capacity for animation or interactive manipulation. Our work introduces a paradigm shift from conventional pixel-based video generation to the creation of high-fidelity avatar with swapped faces. The framework first preprocesses target video to extract FLAME parameters, camera poses and segmentation masks, and then rigs 3D Gaussian splats to the FLAME model across frames, enabling dynamic facial control. To ensure identity preserving, we propose an compound identity embedding constructed from three state-of-the-art face recognition models for avatar finetuning. Finally, we render the face-swapped avatar on the background frames to obtain the face-swapped video. Experimental results demonstrate that GaussianSwap achieves superior identity preservation, visual clarity and temporal consistency, while enabling previously unattainable interactive applications.

Figures (6)

• Figure 1: For video face swapping task, our GaussianSwap can generate not only face-swapped video (4th row) like conventional methods (2nd row) but also face-swapped avatar (3rd row), which can facilitate many interactive applications.
• Figure 2: Overview of the GaussianSwap framework. The framework takes a source image $I_{src}$ and a target video $V_{tgt}$ as input, and generates a high-fidelity avatar from $V_{tgt}$ with the face swapped to match $I_{src}$. In the pipeline, FLAME tracking is first performed on the video sequence $V_{tgt}$ to obtain per-frame FLAME parameters, camera poses and segmentation/matting masks. A 3DGS-based face avatar is then built using the FLAME tracking data, where the 3D Gaussians are dynamically bound to the triangular faces of the FLAME mesh models through 3DGS optimization. To enforce the identity similarity between the avatar and $I_{src}$, the avatar undergoes additional training iterations supervised by three SOTA face recognition models: ArcFace, FaceNet and Dlib. Finally, the high-fidelity, face-swapped avatar is generated, which can be further rendered into the face-swapped video.
• Figure 3: Qualitative results on INSTA. GaussianSwap achieves accurate identity transfer and high-quality visual appearance in the swapped faces. More details are visible in the enlarged view.
• Figure 4: Qualitative results on FF++. GaussianSwap achieves accurate identity transfer and high-quality visual appearance in the swapped faces. More details are visible in the enlarged view.
• Figure 5: Face swapping results for side-view faces on INSTA. GaussianSwap performs effectively on these challenging poses. More details are visible in the enlarged view.