Diffusion-Driven Deceptive Patches: Adversarial Manipulation and Forensic Detection in Facial Identity Verification

TL;DR

This paper addresses the vulnerability of facial identity verification to localized adversarial patches by introducing an end-to-end pipeline that combines FGSM perturbations with diffusion-based refinement to produce highly imperceptible patches. A ViT-GPT2 captioning module provides forensic semantic descriptions of identity on adversarial images, while perceptual hashing and multimodal forensic analysis enable robust detection (SSIM ≈ 0.95). The study presents detailed evaluations of patch effectiveness for identity evasion, caption manipulation, and attack transferability, alongside a comprehensive detection framework leveraging segmentation, heatmaps, and neural-activation maps. The findings highlight significant security risks in biometric systems and offer practical forensic tools for rapid detection and interpretation of adversarial modifications in facial imagery. The methods have implications for security testing, forensic documentation, and the development of defense strategies against patch-based identity and emotion manipulation.

Abstract

This work presents an end-to-end pipeline for generating, refining, and evaluating adversarial patches to compromise facial biometric systems, with applications in forensic analysis and security testing. We utilize FGSM to generate adversarial noise targeting an identity classifier and employ a diffusion model with reverse diffusion to enhance imperceptibility through Gaussian smoothing and adaptive brightness correction, thereby facilitating synthetic adversarial patch evasion. The refined patch is applied to facial images to test its ability to evade recognition systems while maintaining natural visual characteristics. A Vision Transformer (ViT)-GPT2 model generates captions to provide a semantic description of a person's identity for adversarial images, supporting forensic interpretation and documentation for identity evasion and recognition attacks. The pipeline evaluates changes in identity classification, captioning results, and vulnerabilities in facial identity verification and expression recognition under adversarial conditions. We further demonstrate effective detection and analysis of adversarial patches and adversarial samples using perceptual hashing and segmentation, achieving an SSIM of 0.95.

Figures (13)

• Figure 1: Illustrating a ViT-GPT2 based biometric model for status drift captioning and tracking.
• Figure 2: Demonstrating adversarial patch detection pipeline workflow and decision logic.
• Figure 3: Using source CelebA images of 178×218 pixels and adversarial patches of 50×50 pixels, the adversarial attack pipeline demonstrates the interaction between perturbation, diffusion-based patch generation, and identity evasion. This process integrates adversarial noise with patch refinement, using cosine similarity measurements to monitor attack effectiveness report on biometric identity verification system.
• Figure 4: Analysis of biometric labeling in the context of LLM filtration using respected feedback prompts into different categories.
• Figure 5: Demonstrating the influence of conditional prompts in adversarial image generation based on different time steps. We implemented an adversarial attack pipeline that modifies identity classifications by leveraging both FGSM and diffusion-based adversarial sample generation. FGSM is then applied to generate adversarial noise gradually, subtly altering pixel values to mislead the classifier into assigning a different identity label.