DiffPhysBA: Diffusion-based Physical Backdoor Attack against Person Re-Identification in Real-World

TL;DR

This work addresses the security risks of backdoor attacks on person Re-Identification systems by showing how a digital backdoor can be realized in the physical world. It introduces DiffPhysBA, a diffusion-based method that generates realistic semantic triggers (e.g., accessories) guided by ReID features, enabling a training-free transition from digital poisoning to physical activation. The approach achieves strong digital ASR (over 95% on Market-1501) and competitive physical ASR (notably ~94% for Swin-Transformer, with robust results on synthetic data and resilience to certain defenses). The findings highlight a practical vulnerability in ReID systems and call for defenses that address semantic, real-world triggers and domain gaps between digital and physical environments.

Abstract

Person Re-Identification (ReID) systems pose a significant security risk from backdoor attacks, allowing adversaries to evade tracking or impersonate others. Beyond recognizing this issue, we investigate how backdoor attacks can be deployed in real-world scenarios, where a ReID model is typically trained on data collected in the digital domain and then deployed in a physical environment. This attack scenario requires an attack flow that embeds backdoor triggers in the digital domain realistically enough to also activate the buried backdoor in person ReID models in the physical domain. This paper realizes this attack flow by leveraging a diffusion model to generate realistic accessories on pedestrian images (e.g., bags, hats, etc.) as backdoor triggers. However, the noticeable domain gap between the triggers generated by the off-the-shelf diffusion model and their physical counterparts results in a low attack success rate. Therefore, we introduce a novel diffusion-based physical backdoor attack (DiffPhysBA) method that adopts a training-free similarity-guided sampling process to enhance the resemblance between generated and physical triggers. Consequently, DiffPhysBA can generate realistic attributes as semantic-level triggers in the digital domain and provides higher physical ASR compared to the direct paste method by 25.6% on the real-world test set. Through evaluations on newly proposed real-world and synthetic ReID test sets, DiffPhysBA demonstrates an impressive success rate exceeding 90% in both the digital and physical domains. Notably, it excels in digital stealth metrics and can effectively evade state-of-the-art defense methods.

Figures (6)

• Figure 1: The Attack Flow of DiffPhysBA. The adversary is only capable of manipulating the digital training data and injecting the generated purple bag into the benign images. The training data is poisoned in the digital domain, easing the cost of poisoned training data collection. In the inference phase, the adversary does not have access to the digital images captured by the cameras and activates the implanted backdoor by wearing a purple bag, resulting in mismatching of identities.
• Figure 2: Overview of poisoned image generation in the digital domain. Firstly, MMpose is utilized to estimate the pose of the pedestrian and get the bounding box for inpainting. The text encoder generates caption tokens, and the reference image of the bag is encoded by an image encoder. It is then fused with the embedding of the bounding box to create the grounding tokens. Meanwhile, the benign image is fed into the image encoder to obtain visual tokens. These visual tokens are then integrated with the grounding tokens through three distinct attention layers. During the denoising phase, we introduce ReID-driven similarity guidance to enhance the realism of the trigger. To stress that DiffPhysBA does not need to be retrained, the snowflakes denote pre-trained modules.
• Figure 3: The comparison of different attacks against Swin-Transformer on Market-1501 and MSMT17. Since the MSMT17 dataset is not annotated with pedestrian attributes, NaturalNEURIPS2022_8af74993 is compared only on Market-1501.
• Figure 4: The visualization of our attack for the physical and synthetic test sets, respectively. The pedestrians with the highest similarity to the person wearing the physical trigger are the ones specified by the adversary. Please refer to Appendix B\ref{['display_sup']} for more displays.
• Figure 5: Display of generated samples. Each row of (a) displays generated samples from one dataset. In (b), the first column represents the direct pasting of triggers, the triggers in the second and third columns are hats and shoes respectively, while the fourth column shows the samples generated without ReID-driven guidance, and the last column is the samples generated by our method.