DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder

TL;DR

Deepfakes pose challenges for forensic provenance due to blurred identity traces. This paper introduces DFREC, a three-module framework with an Identity Segmentation Module, Source Identity Reconstruction Module, and Target Identity Reconstruction Module, augmented by a Masked Autoencoder to fuse background context with latent target identity features. The overall loss, $L = L_{id} + \lambda_1 L_{perc} + \lambda_2 L_{attr} + \lambda_3 L_{patch}$, guides simultaneous recovery of pristine source and target faces from forgery images. Across FaceForensics++, CelebaMegaFS, and FFHQ-E4S, DFREC achieves superior recovery fidelity and cross-dataset generalization, enabling non-repudiable forensic evidence of provenance by reconstructing both source and target identities directly from forged content.

Abstract

Recent advances in deepfake forensics have primarily focused on improving the classification accuracy and generalization performance. Despite enormous progress in detection accuracy across a wide variety of forgery algorithms, existing algorithms lack intuitive interpretability and identity traceability to help with forensic investigation. In this paper, we introduce a novel DeepFake Identity Recovery scheme (DFREC) to fill this gap. DFREC aims to recover the pair of source and target faces from a deepfake image to facilitate deepfake identity tracing and reduce the risk of deepfake attack. It comprises three key components: an Identity Segmentation Module (ISM), a Source Identity Reconstruction Module (SIRM), and a Target Identity Reconstruction Module (TIRM). The ISM segments the input face into distinct source and target face information, and the SIRM reconstructs the source face and extracts latent target identity features with the segmented source information. The background context and latent target identity features are synergetically fused by a Masked Autoencoder in the TIRM to reconstruct the target face. We evaluate DFREC on six different high-fidelity face-swapping attacks on FaceForensics++, CelebaMegaFS and FFHQ-E4S datasets, which demonstrate its superior recovery performance over state-of-the-art deepfake recovery algorithms. In addition, DFREC is the only scheme that can recover both pristine source and target faces directly from the forgery image with high fadelity.

Figures (8)

• Figure 1: (a) The training process of deepfake face swapping models. The identity loss is used to increase the similarity between the forgery and source face identities, and the attribute loss is used to ensure the similarity between the forgery and target face attributes. (b) The probability density functions of the identity similarity (IDSim) between the forgery and source, forgery and target, and forgery and other unrelated faces. The results are calculated on the five types of forgeries on the FaceForensics++ and CelebaMegaFS datasets.
• Figure 2: An overview of DFREC. The Identity Segmentation Module (ISM) segments an input image to extract the source and target information. The Source Identity Recovery Module (SIRM) and Target Identity Recovery Module (TIRM) recover the source and target identities, respectively.
• Figure 3: Comparison of identity recovery quality of different face inpainting and deepfake recovery methods on face-swapping images of the FaceForensics++ dataset.
• Figure 4: Comparison of identity recovery quality of different face inpainting and deepfake recovery methods on face-swapping images of the CelebaMegaFS dataset.
• Figure 5: Comparison of identity recovery quality of different face inpainting and deepfake recovery methods on face-swapping images of the FFHQ-E4S dataset.