DiffMI: Breaking Face Recognition Privacy via Diffusion-Driven Training-Free Model Inversion

TL;DR

DiffMI leverages a fixed diffusion model to perform training-free model inversion on embedding-based face recognition, addressing limitations of prior GAN- or diffusion-based approaches in generalization and efficiency. Its three-stage latent-code pipeline—robust initialization, top-N selection, and ranked, fine-grained manipulation—coupled with a confidence-aware stopping criterion enables high identity-recovery across unseen identities and models while maintaining cross-model robustness. Empirical evaluations on multiple recognizers and datasets show strong leakage even against inversion-resistant systems, with user studies confirming perceptual identity matches. The results highlight a practical privacy risk in embedding-based frameworks and suggest the need for defenses that go beyond enhancing the training-time or architectural protections of recognition models.

Abstract

Face recognition poses serious privacy risks due to its reliance on sensitive and immutable biometric data. While modern systems mitigate privacy risks by mapping facial images to embeddings (commonly regarded as privacy-preserving), model inversion attacks reveal that identity information can still be recovered, exposing critical vulnerabilities. However, existing attacks are often computationally expensive and lack generalization, especially those requiring target-specific training. Even training-free approaches suffer from limited identity controllability, hindering faithful reconstruction of nuanced or unseen identities. In this work, we propose DiffMI, the first diffusion-driven, training-free model inversion attack. DiffMI introduces a novel pipeline combining robust latent code initialization, a ranked adversarial refinement strategy, and a statistically grounded, confidence-aware optimization objective. DiffMI applies directly to unseen target identities and face recognition models, offering greater adaptability than training-dependent approaches while significantly reducing computational overhead. Our method achieves 84.42%--92.87% attack success rates against inversion-resilient systems and outperforms the best prior training-free GAN-based approach by 4.01%--9.82%. The implementation is available at https://github.com/azrealwang/DiffMI.

Figures (14)

• Figure 1: The threat of model inversion against embedding-based face recognition systems. Although such systems store embeddings instead of raw images for privacy protection, model inversion attacks can reconstruct facial images directly from embeddings, enabling downstream threats such as presentation attacks that can bypass authentication.
• Figure 3: Optimization convergence on the target model and cross-model robustness on the non-target model. To achieve stable cross-model robustness, the optimization must push $\mathcal{L}$ well beyond the target model’s decision threshold $\tau_F$. However, once $\mathcal{L} > \tau_C$, further updates yield diminishing improvements in similarity while degrading cross-model robustness.
• Figure 4: Two-stage latent code generation. First, $K^2$ test $K(\cdot)$d1970transformationd1973testsd1990suggestion filters each sampled latent code $x_G$ based on Gaussian normality, retaining those with $p_K \geq \tau_K$. Then, MTCNN $D(\cdot)$zhang2016joint verifies whether the corresponding image $\hat{x} = G(x_G)$ contains a detectable face, accepting codes with detection confidence $p_D \geq \tau_D$.
• Figure 5: Selection of the top $N$ latent codes based on embedding similarity to the target identity.
• Figure 6: Ranked Adversary algorithm for latent code manipulation. The top $N$ latent codes, ranked in Step (b), are sequentially optimized via adversarial manipulation to maximize the objective in \ref{['eq_loss']}. The process terminates early if any code satisfies $\mathcal{L}_n \geq \tau_C$ for $n = 1, \dots, N$. If none meet the confidence threshold $\tau_C$, the code $x'_{G_n}$ with the highest $\mathcal{L}_n$ is selected, and its reconstruction is returned as the final facial image.