Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shielding Latent Face Representations From Privacy Attacks

Arjun Ramesh Kaushik, Bharat Chandra Yalavarthi, Arun Ross, Vishnu Boddeti, Nalini Ratha

TL;DR

This work addresses the privacy risks of face embeddings leaking soft biometrics by proposing a multi-layer shield that combines embedding compression (MRL), Fully Homomorphic Encryption (FHE), and an irreversible feature manifold hash (PolyProtect base). The method enables encrypted computation for face recognition while suppressing sensitive attributes such as age, gender, and ethnicity, and maintains high biometric utility as demonstrated on two encoders and two datasets. Differential Privacy and existing template-protection techniques are evaluated and found insufficient alone, motivating the layered approach; experiments show near-random classification for soft-biometric attributes under the encrypted framework and substantial privacy gains with modest impact on identity accuracy. The approach leverages FHE theoretical guarantees and an additional irreversible transform to remain protective even if a secret key is compromised, offering a practical privacy-preserving pathway for cloud-based facial analytics. Overall, the paper demonstrates a significant advancement in balancing privacy and utility in embeddings through a rigorously tested, multi-layer strategy.

Abstract

In today's data-driven analytics landscape, deep learning has become a powerful tool, with latent representations, known as embeddings, playing a central role in several applications. In the face analytics domain, such embeddings are commonly used for biometric recognition (e.g., face identification). However, these embeddings, or templates, can inadvertently expose sensitive attributes such as age, gender, and ethnicity. Leaking such information can compromise personal privacy and affect civil liberty and human rights. To address these concerns, we introduce a multi-layer protection framework for embeddings. It consists of a sequence of operations: (a) encrypting embeddings using Fully Homomorphic Encryption (FHE), and (b) hashing them using irreversible feature manifold hashing. Unlike conventional encryption methods, FHE enables computations directly on encrypted data, allowing downstream analytics while maintaining strong privacy guarantees. To reduce the overhead of encrypted processing, we employ embedding compression. Our proposed method shields latent representations of sensitive data from leaking private attributes (such as age and gender) while retaining essential functional capabilities (such as face identification). Extensive experiments on two datasets using two face encoders demonstrate that our approach outperforms several state-of-the-art privacy protection methods.

Shielding Latent Face Representations From Privacy Attacks

TL;DR

This work addresses the privacy risks of face embeddings leaking soft biometrics by proposing a multi-layer shield that combines embedding compression (MRL), Fully Homomorphic Encryption (FHE), and an irreversible feature manifold hash (PolyProtect base). The method enables encrypted computation for face recognition while suppressing sensitive attributes such as age, gender, and ethnicity, and maintains high biometric utility as demonstrated on two encoders and two datasets. Differential Privacy and existing template-protection techniques are evaluated and found insufficient alone, motivating the layered approach; experiments show near-random classification for soft-biometric attributes under the encrypted framework and substantial privacy gains with modest impact on identity accuracy. The approach leverages FHE theoretical guarantees and an additional irreversible transform to remain protective even if a secret key is compromised, offering a practical privacy-preserving pathway for cloud-based facial analytics. Overall, the paper demonstrates a significant advancement in balancing privacy and utility in embeddings through a rigorously tested, multi-layer strategy.

Abstract

In today's data-driven analytics landscape, deep learning has become a powerful tool, with latent representations, known as embeddings, playing a central role in several applications. In the face analytics domain, such embeddings are commonly used for biometric recognition (e.g., face identification). However, these embeddings, or templates, can inadvertently expose sensitive attributes such as age, gender, and ethnicity. Leaking such information can compromise personal privacy and affect civil liberty and human rights. To address these concerns, we introduce a multi-layer protection framework for embeddings. It consists of a sequence of operations: (a) encrypting embeddings using Fully Homomorphic Encryption (FHE), and (b) hashing them using irreversible feature manifold hashing. Unlike conventional encryption methods, FHE enables computations directly on encrypted data, allowing downstream analytics while maintaining strong privacy guarantees. To reduce the overhead of encrypted processing, we employ embedding compression. Our proposed method shields latent representations of sensitive data from leaking private attributes (such as age and gender) while retaining essential functional capabilities (such as face identification). Extensive experiments on two datasets using two face encoders demonstrate that our approach outperforms several state-of-the-art privacy protection methods.
Paper Structure (16 sections, 6 figures, 7 tables)

This paper contains 16 sections, 6 figures, 7 tables.

Table of Contents

  1. INTRODUCTION
  2. Threat Model
  3. FHE Basics
  4. Prior Work
  5. Proposed Approach
  6. Experiments
  7. Datasets
  8. Irreversible Feature Manifold Hash techniques
  9. Embedding Compression
  10. Differential Privacy
  11. Fully Homomorphic Encryption (FHE)
  12. Execution Times
  13. Results
  14. Analysis
  15. Conclusion
  16. ...and 1 more sections

Figures (6)

  • Figure 1: (a) Existing embedding protection techniques typically guarantee unlinkability and irreversibility but leak sensitive information like age, gender, and ethnicity. (b) Our proposed method improves on the current methods by shielding the sensitive information against privacy attacks while still ensuring unlinkability and irreversibility. The face image used in the figure is AI-generated.
  • Figure 2: Ablation Study with PolyProtect shows soft biometric leakage under different settings. (a) Overlap (b) m - Length of polynomial coefficients/exponents (c) [-C,C] - Range of polynomial coefficients.
  • Figure 3: Performance of attribute extraction (Age, Gender, Ethnicity), along with Rank-1 identification accuracy (Identity), after compression of embeddings using Matryoshka Representation Learning (MRL), as a function of different compression dimensions. The graph is based on an experiment performed using AdaFace on the CelebSet dataset.
  • Figure 4: Classification accuracy of attributes (Age, Gender, Ethnicity) along with Rank-1 identification accuracy (Identity) of face embeddings protected by Differential Privacy as a function of Privacy Budget ($\epsilon$) and Sensitivity ($\Delta f = 2$). The graph is based on an experiment performed using AdaFace on the CelebSet dataset.
  • Figure 5: False Match Rate (FMR) vs False Non-Match Rate (FNMR) for identity verification performance using unprotected and privacy-protected embeddings - (a) AdaFace - CFD; (b) AdaFace - CelebSet; (c) ArcFace - CFD; (d) ArcFace - CelebSet. A smaller area-under-curve signifies better verification performance. PP curve overlaps the "unprotected embeddings" curve, showing a negligible loss in identity verification performance.
  • ...and 1 more figures