Adaptive Hybrid Masking Strategy for Privacy-Preserving Face Recognition Against Model Inversion Attack

TL;DR

The paper tackles privacy leakage in face recognition (FR) models caused by model inversion attacks (MIA). It introduces Adaptive Hybrid Masking, combining frequency-domain masking (PPFR-FD) with reinforcement-learning–driven MixUp in the frequency domain, where a strategy network selects $k$ from ${2,3,4,5,6}$ to maximize FR loss as a privacy reward while the FR network minimizes its loss. A formal loss objective blends $L_0$ (FR loss on mixed data) with RL-derived rewards $L_1$ and $L_2$, solved via alternating projection and REINFORCE, to foster an adversarial but balanced training dynamic. The authors propose a comprehensive evaluation framework (S1–S4, Score_{pp}, Score_{cls}) and demonstrate that the proposed method improves privacy protection with minimal impact on recognition accuracy in centralized experiments, including robustness against CGAN-based white-box and black-box attacks.

Abstract

The utilization of personal sensitive data in training face recognition (FR) models poses significant privacy concerns, as adversaries can employ model inversion attacks (MIA) to infer the original training data. Existing defense methods, such as data augmentation and differential privacy, have been employed to mitigate this issue. However, these methods often fail to strike an optimal balance between privacy and accuracy. To address this limitation, this paper introduces an adaptive hybrid masking algorithm against MIA. Specifically, face images are masked in the frequency domain using an adaptive MixUp strategy. Unlike the traditional MixUp algorithm, which is predominantly used for data augmentation, our modified approach incorporates frequency domain mixing. Previous studies have shown that increasing the number of images mixed in MixUp can enhance privacy preservation but at the expense of reduced face recognition accuracy. To overcome this trade-off, we develop an enhanced adaptive MixUp strategy based on reinforcement learning, which enables us to mix a larger number of images while maintaining satisfactory recognition accuracy. To optimize privacy protection, we propose maximizing the reward function (i.e., the loss function of the FR system) during the training of the strategy network. While the loss function of the FR network is minimized in the phase of training the FR network. The strategy network and the face recognition network can be viewed as antagonistic entities in the training process, ultimately reaching a more balanced trade-off. Experimental results demonstrate that our proposed hybrid masking scheme outperforms existing defense algorithms in terms of privacy preservation and recognition accuracy against MIA.

Figures (3)

• Figure 1: Comparison of the results of MIA zhang2020secret and accuracy for different defense methods. From the left to right, we show the original image, the result of no defense, Mixup, Instahide DP and our denfense methods. The value below each image indicates recognition accuracy of the corresponding model. We expect to obtain a high-accuracy performance model while its result of MIA looks as much as different from the original image. Some existing defense methods cannot achieve a better trade-off between privacy and accuracy.
• Figure 2: The framework of (a) our proposed defense method and (b) the strategy network.
• Figure 3: Face images recovered by CGAN for different masking methods. Results from the 1st row to the last row correspond to the original images, Masking+AdaMixUp(k:2-4), Masking+AdaMixUp(k:2-5), Masking+AdaMixUp(k:2-6), MixUp(k=2), PPFR-FD(Masking), DP, and Instahide.