Modality Unified Attack for Omni-Modality Person Re-Identification

TL;DR

The paper addresses the vulnerability of omni-modality person re-id models to adversarial examples by introducing a Modality Unified Attack (MUA) that trains modality-specific generators using a multi-modality surrogate. It combines three disruption mechanisms—Metric Disruption (MD), Cross Modality Simulated Disruption (CMSD), and Multi Modality Collaborative Disruption (MMCD)—to degrade common and cross-modality feature embeddings and robustly attack single-, cross-, and multi-modality models. The approach demonstrates strong transferability across diverse targets and shows effectiveness against defenses, establishing a new framework for evaluating and improving robustness in omni-modality re-id. The findings highlight practical security risks in surveillance systems and provide a foundation for adversarial training and defense strategies to enhance reliability in real-world deployments.

Abstract

Deep learning based person re-identification (re-id) models have been widely employed in surveillance systems. Recent studies have demonstrated that black-box single-modality and cross-modality re-id models are vulnerable to adversarial examples (AEs), leaving the robustness of multi-modality re-id models unexplored. Due to the lack of knowledge about the specific type of model deployed in the target black-box surveillance system, we aim to generate modality unified AEs for omni-modality (single-, cross- and multi-modality) re-id models. Specifically, we propose a novel Modality Unified Attack method to train modality-specific adversarial generators to generate AEs that effectively attack different omni-modality models. A multi-modality model is adopted as the surrogate model, wherein the features of each modality are perturbed by metric disruption loss before fusion. To collapse the common features of omni-modality models, Cross Modality Simulated Disruption approach is introduced to mimic the cross-modality feature embeddings by intentionally feeding images to non-corresponding modality-specific subnetworks of the surrogate model. Moreover, Multi Modality Collaborative Disruption strategy is devised to facilitate the attacker to comprehensively corrupt the informative content of person images by leveraging a multi modality feature collaborative metric disruption loss. Extensive experiments show that our MUA method can effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%, 49.0% and 62.7% mean mAP Drop Rate, respectively.

Figures (7)

• Figure 1: The diagram of modality unified adversarial attack for omni-modality (single-, cross- and multi-modality) person re-id models. $G_{R}$, $G_{N}$ and $G_{T}$ are modality unified adversarial generators for RGB, NI and TI images, which can generate AEs to successfully attack all three types of re-id models, e.g., the RGB modality unified adversarial generator $G_{R}$ can generate RGB AEs that simultaneously mislead single-, cross- and multi-modality re-id models.
• Figure 2: The overall framework of our proposed MUA. The multi-modality model is adopted as surrogate model, and features $\mathcal{F}_h$ extracted by subnetworks $\mathcal{S}_h$ are perturbed before fusion operation. Adversarial attack generator $\mathcal{G}_h$ and discriminator $\mathcal{D}_h$ are trained in GAN scheme to obtain imperceptible and qualify AEs. Metric Disruption, CMSD and MMCD constraints guide the adversarial generator produce transferable AEs across omni-modality re-id models.
• Figure 3: The comparisons of MD and MMCD. The blue, green and red regions represent distinct feature spaces corresponding to RGB, NI and TI modality images of the same person. The regions where different modality feature spaces intersect are shared feature subspaces across different modalities. $R_c$, $N_c$ and $T_c$ are individual feature centers for each modality. Adversarial features under MD constraints may only move away from the original features and cannot ensure an effective departure away from the individual feature space, whereas under the guidance of MMCD, the adversarial features are constrained to directly far away from the individual feature space.
• Figure 4: Attention maps of RGB, NI, and TI modalities for both benign and adversarial examples across different re-ID models. The original attention maps correspond to benign images, while the others visualize adversarial examples generated under different loss constraints (MD, MMCD, CMSD). Transreid he2021transreid, MMN zhang2021towards and EDITOR zhang2024magic are token as the single-, cross- and multi-modality re-id models for Visualizations.
• Figure 5: Analysis of mDR under different perturbation strength and balance weights. The mDR values in the balance weights analysis represent the average across four retrieval settings.