Federated Face Forgery Detection Learning with Personalized Representation

TL;DR

The paper tackles privacy-preserving face forgery detection under non-IID data by introducing FedPR, a personalized federated learning framework. FedPR decomposes each client’s model into a personalized and a shared representation, using adaptive feature-statistic operations $R_p$ and $R_s$ and optimizing a composite loss $L = \alpha L_{adv} + \beta L_p + \gamma L_s$ to encourage robust, client-specific cues while enabling server-side sharing of common representations. The approach is validated on multiple public datasets and a Forgery Source Hybrid dataset, showing superior performance and demonstrating the tangible benefits of personalized representations for cross-dataset generalization, with ablations confirming the contribution of personalization. The work preserves data privacy by keeping private data on-device and only sharing the learned shared representations, achieving strong detection performance across diverse forgery types and suggesting extensions to cross-modal forgery detection and other domains. Overall, FedPR represents a significant advancement in privacy-aware, robust forgery detection with practical implications for real-world deployment.

Abstract

Deep generator technology can produce high-quality fake videos that are indistinguishable, posing a serious social threat. Traditional forgery detection methods directly centralized training on data and lacked consideration of information sharing in non-public video data scenarios and data privacy. Naturally, the federated learning strategy can be applied for privacy protection, which aggregates model parameters of clients but not original data. However, simple federated learning can't achieve satisfactory performance because of poor generalization capabilities for the real hybrid-domain forgery dataset. To solve the problem, the paper proposes a novel federated face forgery detection learning with personalized representation. The designed Personalized Forgery Representation Learning aims to learn the personalized representation of each client to improve the detection performance of individual client models. In addition, a personalized federated learning training strategy is utilized to update the parameters of the distributed detection model. Here collaborative training is conducted on multiple distributed client devices, and shared representations of these client models are uploaded to the server side for aggregation. Experiments on several public face forgery detection datasets demonstrate the superior performance of the proposed algorithm compared with state-of-the-art methods. The code is available at \emph{https://github.com/GANG370/PFR-Forgery.}

Figures (5)

• Figure 1: The differences between our method and traditional face forgery detection methods and traditional federated learning forgery detection method. (a) Traditional forgery detection methods require centralized aggregation of all client data for training, which is detrimental to privacy protection. (b) Traditional federated learning forgery detection method requires uploading all client model parameters to the server side, which prevents learning unique representations from individual clients. (c) Our method can extract personalized representations for complex forgery datasets with diverse types, and upload the shared representation to the server side for updates. In the testing stage, each client leverages its personalized model for local testing.
• Figure 2: The overall framework of the proposed federated face forgery detection learning with personalized representation method.
• Figure 3: The samples of public face forgery datasets: (a) FaceForensics++. (b)WildDeepfake. (c) CelebDF-v2. (d) Deeperforensics-1.0. (e) FMFCC-V.
• Figure 4: Results of cross-validation on different training and testing subsets. The vertical axis represents training data, the horizontal axis represents testing data, and the evaluation index is accuracy.
• Figure 5: Cross-validation results on different training and testing subsets of the Forgery Source Hybrid Dataset without personalized federated learning. The vertical axis represents training data, the horizontal axis represents testing data, and the evaluation index is accuracy.