Machine Pareidolia: Protecting Facial Image with Emotional Editing

TL;DR

A pioneering approach that employs human emotion modifications to disguise original identities as target identities in facial images, which surpasses previous baselines, including noise-based, makeup-based, and freeform attribute methods in both qualitative fidelity and quantitative metrics.

Abstract

The proliferation of facial recognition (FR) systems has raised privacy concerns in the digital realm, as malicious uses of FR models pose a significant threat. Traditional countermeasures, such as makeup style transfer, have suffered from low transferability in black-box settings and limited applicability across various demographic groups, including males and individuals with darker skin tones. To address these challenges, we introduce a novel facial privacy protection method, dubbed \textbf{MAP}, a pioneering approach that employs human emotion modifications to disguise original identities as target identities in facial images. Our method uniquely fine-tunes a score network to learn dual objectives, target identity and human expression, which are jointly optimized through gradient projection to ensure convergence at a shared local optimum. Additionally, we enhance the perceptual quality of protected images by applying local smoothness regularization and optimizing the score matching loss within our network. Empirical experiments demonstrate that our innovative approach surpasses previous baselines, including noise-based, makeup-based, and freeform attribute methods, in both qualitative fidelity and quantitative metrics. Furthermore, MAP proves its effectiveness against an online FR API and shows advanced adaptability in uncommon photographic scenarios.

Figures (9)

• Figure 1: Comparison of makeup-based baselines with our MAP. Baseline methods (top) use makeup styles obscure original identities, optimizing objectives independently, which reduces efficiency and limits applicability across demographics (e.g., males). In contrast, MAP (bottom) leverages human emotion modifications and a unified optimization strategy to disguise original identities as target identities, achieving universal robustness.
• Figure 2: Comparison between makeup transfer-based and emotion-based approaches in terms of frequency changes (with azimuthal integral). Makeup transfer primarily edits the image in the low-frequency range, while our emotion-based approach targets medium to high frequencies, resulting in a more natural appearance and being better suited for obfuscating identity adversarial noises.
• Figure 3: Illustration of our dual objectives optimization strategies. Naively optimizing unrelated tasks (identity $\boldsymbol{\leftarrow}$ and emotion $\boldsymbol{\rightarrow}$) may lead to negative transfer, canceling out each other's gradients. Our approach can render a new update ($\boldsymbol{\uparrow}$) that helps guide the model towards optimal values.
• Figure 4: Top: Illustration of Laplacian Smoothness Regularization for landmarks (pzd72) 54$^{th}$ and 26$^{th}$. $\blacktriangle$ represents the average of neighbors ($\Large \bullet$) of pzd72. Bottom: Effects of Laplacian Smoothness; without it, the eyebrow is eroded compared to the original image.
• Figure 5: Visualizations of protected face images generated by different facial privacy protection methods on CelebA-HQ. The yellow numbers in each image represent confidence scores returned by Face++. Unlike makeup-based approaches, which may not be suitable for all demographics, our method successfully protects images against malicious FR systems through emotion editing (top to bottom: surprise, surprise, happy), while preserving original details like color grading and background.