Prompt-Guided Environmentally Consistent Adversarial Patch

TL;DR

This paper proposes Prompt-Guided Environmentally Consistent Adversarial Patch (PG-ECAP), a method that aligns the patch with the environment to ensure seamless integration into the environment and introduces two alignment losses to enhance the naturalness and consistency of the patches.

Abstract

Adversarial attacks in the physical world pose a significant threat to the security of vision-based systems, such as facial recognition and autonomous driving. Existing adversarial patch methods primarily focus on improving attack performance, but they often produce patches that are easily detectable by humans and struggle to achieve environmental consistency, i.e., blending patches into the environment. This paper introduces a novel approach for generating adversarial patches, which addresses both the visual naturalness and environmental consistency of the patches. We propose Prompt-Guided Environmentally Consistent Adversarial Patch (PG-ECAP), a method that aligns the patch with the environment to ensure seamless integration into the environment. The approach leverages diffusion models to generate patches that are both environmental consistency and effective in evading detection. To further enhance the naturalness and consistency, we introduce two alignment losses: Prompt Alignment Loss and Latent Space Alignment Loss, ensuring that the generated patch maintains its adversarial properties while fitting naturally within its environment. Extensive experiments in both digital and physical domains demonstrate that PG-ECAP outperforms existing methods in attack success rate and environmental consistency.

Figures (6)

• Figure 1: A comparison of various adversarial clothes: (a) Adversarial T-shirt xu2020adversarial, (b) Adversarial Texture hu2022adversarial, (c) NAP hu2021naturalistic, (d) DAP guesmi2024dap, and (e) Ours. We cover our generated adversarial patch onto the long-sleeved T-shirt to create our adversarial clothing. Among these methods, only our clothing consistent with the environment, making it more practical in real-world scenarios.
• Figure 2: Ours vs State-of-the-Art patches: (a) AdvYolo thys2019fooling, (b) AdvTexture hu2022adversarial, (c) T-SEA huang2023t, (d) AdvT-shirt xu2020adversarial, (e) UPC huang2020universal, (f) NAP hu2021naturalistic, (g) DAP guesmi2024dap, (h) AdvCat hu2023physically, and (i) Ours. Our patches achieve a more natural and environmentally consistent appearance in forest-like environments.
• Figure 3: An overview of the proposed PG-ECAP. We first extract prompt $\mathcal{P}$ from the environment and feed $\mathcal{P}$ into a text encoder $\psi(.)$ to obtain text embeddings $\mathcal{C}$. Then, we sample $z_T$ from a Gaussian distribution and feed it with $\mathcal{C}$ into the diffusion model, extracting the cross attention maps during the DDIM process for alignment. After the DDIM process is finished, we align both the cross attention maps and $z_0$ with their corresponding initial values to ensure that the generated image aligns with $\mathcal{P}$. Finally, we decode the optimized $z_0$ to obtain the adversarial patch, augment it using EOT, and attach them onto the images to generate adversarial inputs. These inputs are then fed into the model to obtain detection confidence scores, which serve as the loss function to optimize $z_T$ with the two alignment losses.
• Figure 4: During the optimization of $z_T$, without constraints, the adversarial patch gradually misaligns with $\mathcal{P}$, where $\mathcal{P}$ is "a picture full of leaf-like green color".
• Figure 5: The detection results of four postures in four scenes. Our generated clothing can successfully evade detection in different scenes with different postures.