Facial Expression Recognition with Controlled Privacy Preservation and Feature Compensation

TL;DR

This paper tackles privacy concerns in video-based facial expression recognition (FER) by introducing a dual-frequency privacy-preservation framework that uses a wavelet transform to separately remove identity features from high- and low-frequency components. It decouples privacy and utility tasks with a privacy-enhancement controller per frequency and a feature compensator that enriches FER-relevant information, plus a privacy leakage validator to quantify residual identity information. On CREMA-D, the approach achieves a FER accuracy of $78.84\%$ with a privacy leakage ratio of $2.01\%$, outperforming several baselines including Gaussian blur, optical flow, and image swapping, while maintaining robust FER performance. The work provides a practical mechanism for quantifying privacy leakage in closed-set FER and demonstrates a scalable pathway toward secure, video-based FER applications in privacy-sensitive settings.

Abstract

Facial expression recognition (FER) systems raise significant privacy concerns due to the potential exposure of sensitive identity information. This paper presents a study on removing identity information while preserving FER capabilities. Drawing on the observation that low-frequency components predominantly contain identity information and high-frequency components capture expression, we propose a novel two-stream framework that applies privacy enhancement to each component separately. We introduce a controlled privacy enhancement mechanism to optimize performance and a feature compensator to enhance task-relevant features without compromising privacy. Furthermore, we propose a novel privacy-utility trade-off, providing a quantifiable measure of privacy preservation efficacy in closed-set FER tasks. Extensive experiments on the benchmark CREMA-D dataset demonstrate that our framework achieves 78.84% recognition accuracy with a privacy (facial identity) leakage ratio of only 2.01%, highlighting its potential for secure and reliable video-based FER applications.

Figures (4)

• Figure 1: Our framework: (a). Controlled privacy-preservation in the high- and low-frequencies (Freq.) with (b). Controlled feature compensation and (c). Privacy leakage validation. The results show the performance of different privacy preservation approaches with video-based FER on privacy leakage ratio and video FER accuracy. GB 1 & 2, OF, Trade-off and IS are short for Gaussian Blurring 1 & 2, Optical Flow, Trade-off framework and Image Swapping, respectively.
• Figure 2: Illustration of our framework. 1 Controlled privacy-preservation: the original video ($V_i$) is transformed into high- and low-frequency components ($V_{ih}$ and $V_{il}$), followed by privacy enhancement, which includes a privacy enhancer ($F_{hpr}$ and $F_{lpr}$) and its controller to strengthen privacy enhancement ($C_{hpr}$ and $C_{lpr}$) for low- and high-frequency components respectively. The privacy-preserved high- and low-frequency frames are then combined and inverse-transformed into new video frames ($G_i$) for the next step. 2 Controlled feature compensation involves a feature compensator ($F_{fc}$) and its controller ($C_{fc}$), which regulates the compensation of specific features. 3 Privacy leakage validation uses a privacy leakage validator ($V_{pl}$) to determine the proportion of identity features that can be recognized to measure the performance of privacy preservation. 4 Utility task ($F_{u}$) learns from the feature-rich video frames ($C_i$))on its main task, such as FER in our case. All grey components are frozen during training, and all controllers are not involved in inference.
• Figure 3: $F_{pr}$ and $C_{pr}$ denote the privacy enhancer and its controller. They work on the original video. The privacy enhancer and identity budget controller (in purple) share identical architectures and weights with $F_{hpr}$ and $C_{hpr}$, respectively. In Task 2, the controller-free privacy enhancer is trained while updating $F_{u}$.
• Figure 4: The structure of Tasks 4, 5 and 6. In Task 6, the parameters of controller-free $F_{hpr}$ and $F_{lpr}$ are updated while training $F_{u}$.