AlphaFace: High Fidelity and Real-time Face Swapper Robust to Facial Pose

TL;DR

AlphaFace addresses the challenge of robust, real-time face swapping under extreme facial poses without relying on explicit geometric priors. It introduces a vision-language supervision pipeline using CLIP-based image and text losses, coupled with a cross-adaptive identity injection (CAII) module to preserve identity while maintaining target pose and expression. Empirical results on FF++, MPIE, and LPFF show state-of-the-art pose and expression accuracy with competitive identity fidelity and real-time performance (~41 FPS). The approach demonstrates how semantic supervision from a VLM can enhance identity swapping robustness and fidelity in dynamic, pose-diverse scenarios, with public code release enabling broader adoption.

Abstract

Existing face-swapping methods often deliver competitive results in constrained settings but exhibit substantial quality degradation when handling extreme facial poses. To improve facial pose robustness, explicit geometric features are applied, but this approach remains problematic since it introduces additional dependencies and increases computational cost. Diffusion-based methods have achieved remarkable results; however, they are impractical for real-time processing. We introduce AlphaFace, which leverages an open-source vision-language model and CLIP image and text embeddings to apply novel visual and textual semantic contrastive losses. AlphaFace enables stronger identity representation and more precise attribute preservation, all while maintaining real-time performance. Comprehensive experiments across FF++, MPIE, and LPFF demonstrate that AlphaFace surpasses state-of-the-art methods in pose-challenging cases. The project is publicly available on `https://github.com/andrewyu90/Alphaface_Official.git'.

Figures (10)

• Figure 1: Examples of the results of face identity swapping on various facial poses obtained by AlphaFace and recent SOTA methods based on diffusion model zhao2023diffswap and exploit explicit geometric features rosberg2023facedancerwang2021hififace. Compared to the frontal face image, the swapped results of the SOTA methods for extreme poses (greater than $\pm$45 degrees) remain highly distorted.
• Figure 2: The detailed information for the architecture and workflow of AlphaFace. (a) illustrates the workflow details for training and testing of AlphaFace. (b) shows the architectural details of the cross-adaptive identity injection (CAII) block. The red, blue, and green arrow lines define the pipelines for source $x_{\text{s}}$, target $x_{\text{t}}$, and swapped face $x_{\text{t}\rightarrow{}\text{s}}$ images for training, respectively. The purple dotted lines define the pipeline to generate $x_{\text{t}\rightarrow{}\text{s}}$.
• Figure 3: Qualitative results of AlphaFace and the existing SOTA methods shiohara2023blendfacenirkin2019fsganchen2020simswapwang2021hififacerosberg2023facedancerzhao2023diffswap on FF++ dataset rossler2019faceforensics++. The extended results are shown in Appendix \ref{['appx:ff++_results']}.
• Figure 4: Face swapping results of the AlphaFace and other existing SOTA methods zhao2023diffswapwang2021hififacerosberg2023facedancershiohara2023blendfacechen2020simswapnirkin2019fsgan on the LPFF dataset wu2023lpff. (a) and (b) represents the swapping results on rotated and tilted facial pose cases, respectively. Extended results are provided in Appendix \ref{['appex:lpff_extended']}.
• Figure 5: The face identity swapping results of the AlphaFace and other methods wang2021hififacerosberg2023facedancerzhao2023diffswap on the MPIE dataset gross2010multi. The extended results are shown in Appendix \ref{['appx:mpie_results']}.