Privacy-Preserving Face Recognition in Hybrid Frequency-Color Domain
Dong Han, Yong Li, Joachim Denzler
TL;DR
The paper addresses privacy in face recognition by reducing sensitivity of both inputs and embeddings. It introduces a hybrid frequency-color fusion (HFCF) pipeline that converts RGB inputs into a compact frequency representation (BDCT) with sparse color cues (DLBP/LBP) and applies an identity-specific embedding mapping, protected by secure multiparty computation for similarity evaluation. Key innovations include cross-channel BDCT fusion to 63 channels, color descriptors that preserve privacy, and PolyProtect-inspired embedding protection coupled with SMPC for secure 1:N inference. Empirical results on large-scale verification tasks show substantial accuracy gains over frequency-based baselines (approximately 2.6% to 4.2% in 1:N) while enhancing privacy for inputs and embeddings, indicating strong practical potential for privacy-preserving FR systems.
Abstract
Face recognition technology has been deployed in various real-life applications. The most sophisticated deep learning-based face recognition systems rely on training millions of face images through complex deep neural networks to achieve high accuracy. It is quite common for clients to upload face images to the service provider in order to access the model inference. However, the face image is a type of sensitive biometric attribute tied to the identity information of each user. Directly exposing the raw face image to the service provider poses a threat to the user's privacy. Current privacy-preserving approaches to face recognition focus on either concealing visual information on model input or protecting model output face embedding. The noticeable drop in recognition accuracy is a pitfall for most methods. This paper proposes a hybrid frequency-color fusion approach to reduce the input dimensionality of face recognition in the frequency domain. Moreover, sparse color information is also introduced to alleviate significant accuracy degradation after adding differential privacy noise. Besides, an identity-specific embedding mapping scheme is applied to protect original face embedding by enlarging the distance among identities. Lastly, secure multiparty computation is implemented for safely computing the embedding distance during model inference. The proposed method performs well on multiple widely used verification datasets. Moreover, it has around 2.6% to 4.2% higher accuracy than the state-of-the-art in the 1:N verification scenario.