Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards an End-to-End (E2E) Adversarial Learning and Application in the Physical World

Dudi Biton, Jacob Shams, Satoru Koda, Asaf Shabtai, Yuval Elovici, Ben Nassi

TL;DR

This work tackles the transferability gap that arises when adversarial patches learned in the digital domain are applied in the physical world. It introduces PAPLA, an end-to-end physical-domain adversarial patch learning framework that uses a projector to generate and apply patches directly on the target in real environments. Through controlled and outdoor experiments, PAPLA demonstrates improved evasion of object detectors across multiple targets, surfaces, and detectors by integrating environmental factors into the learning loop. The study analyzes environmental dependencies, image-distortion metrics, and cross-detector transferability, highlighting both the practical potential and limitations of physical-domain adversarial learning for real-world adversarial scenarios.

Abstract

The traditional learning process of patch-based adversarial attacks, conducted in the digital domain and then applied in the physical domain (e.g., via printed stickers), may suffer from reduced performance due to adversarial patches' limited transferability from the digital domain to the physical domain. Given that previous studies have considered using projectors to apply adversarial attacks, we raise the following question: can adversarial learning (i.e., patch generation) be performed entirely in the physical domain with a projector? In this work, we propose the Physical-domain Adversarial Patch Learning Augmentation (PAPLA) framework, a novel end-to-end (E2E) framework that converts adversarial learning from the digital domain to the physical domain using a projector. We evaluate PAPLA across multiple scenarios, including controlled laboratory settings and realistic outdoor environments, demonstrating its ability to ensure attack success compared to conventional digital learning-physical application (DL-PA) methods. We also analyze the impact of environmental factors, such as projection surface color, projector strength, ambient light, distance, and angle of the target object relative to the camera, on the effectiveness of projected patches. Finally, we demonstrate the feasibility of the attack against a parked car and a stop sign in a real-world outdoor environment. Our results show that under specific conditions, E2E adversarial learning in the physical domain eliminates the transferability issue and ensures evasion by object detectors. Finally, we provide insights into the challenges and opportunities of applying adversarial learning in the physical domain and explain where such an approach is more effective than using a sticker.

Towards an End-to-End (E2E) Adversarial Learning and Application in the Physical World

TL;DR

This work tackles the transferability gap that arises when adversarial patches learned in the digital domain are applied in the physical world. It introduces PAPLA, an end-to-end physical-domain adversarial patch learning framework that uses a projector to generate and apply patches directly on the target in real environments. Through controlled and outdoor experiments, PAPLA demonstrates improved evasion of object detectors across multiple targets, surfaces, and detectors by integrating environmental factors into the learning loop. The study analyzes environmental dependencies, image-distortion metrics, and cross-detector transferability, highlighting both the practical potential and limitations of physical-domain adversarial learning for real-world adversarial scenarios.

Abstract

The traditional learning process of patch-based adversarial attacks, conducted in the digital domain and then applied in the physical domain (e.g., via printed stickers), may suffer from reduced performance due to adversarial patches' limited transferability from the digital domain to the physical domain. Given that previous studies have considered using projectors to apply adversarial attacks, we raise the following question: can adversarial learning (i.e., patch generation) be performed entirely in the physical domain with a projector? In this work, we propose the Physical-domain Adversarial Patch Learning Augmentation (PAPLA) framework, a novel end-to-end (E2E) framework that converts adversarial learning from the digital domain to the physical domain using a projector. We evaluate PAPLA across multiple scenarios, including controlled laboratory settings and realistic outdoor environments, demonstrating its ability to ensure attack success compared to conventional digital learning-physical application (DL-PA) methods. We also analyze the impact of environmental factors, such as projection surface color, projector strength, ambient light, distance, and angle of the target object relative to the camera, on the effectiveness of projected patches. Finally, we demonstrate the feasibility of the attack against a parked car and a stop sign in a real-world outdoor environment. Our results show that under specific conditions, E2E adversarial learning in the physical domain eliminates the transferability issue and ensures evasion by object detectors. Finally, we provide insights into the challenges and opportunities of applying adversarial learning in the physical domain and explain where such an approach is more effective than using a sticker.
Paper Structure (38 sections, 9 figures, 5 tables)

This paper contains 38 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Transferability Between Digital and Physical Domains
  3. Experimental Setup
  4. Failure of Transferability
  5. Cause of the Failure
  6. Differences Between Digital and Physical Patches
  7. Differences Between Consecutive Captures of the Same Scene
  8. Threat Model and Method
  9. Threat Model
  10. Attacker's Capabilities and Knowledge
  11. Extension of Previously Evaluated Threat Model
  12. Significance
  13. Method - PAPLA
  14. Analysis
  15. Impact of Environmental Factors on Attack Success
  16. ...and 23 more sections

Figures (9)

  • Figure 1: Application of adversarial patches in different learning scenarios. We applied the NAP hu2021naturalistic attack against the Faster R-CNN object detector in four different scenarios: (a) no application of NAP, (b) the adversarial patch was generated and applied to the object in the digital domain, (c) the patch was generated digitally and physically applied to the object as a sticker, and (d) using PAPLA, our E2E framework, the patch was generated and applied in the physical domain, causing the object detector to fail to detect the cup.
  • Figure 2: PAPLA learning process: an adversary points a projector and a camera at the target object and (1) projects a patch onto the object, as well as (2) captures the scene that contains the object with the projected patch. (3) The patch pixels are updated using PAPLA, and the process repeats.
  • Figure 3: Confidence reduction percentage for different angles, distances, ambient light levels, and projectors. Each cell shows the percentage difference between the original confidence score (without patch projection) and the confidence score with patch projection learned E2E in the physical domain.
  • Figure 4: Box plots illustrating the impact of each environmental factor on the confidence reduction percentage of the DPatch attack performed using PAPLA (E2E in the physical domain). The Y-axis represents the percentage difference between the original confidence score (without patch projection) and the confidence score with patch projection using PAPLA.
  • Figure 5: Impact of surface color on patch projection effectiveness: Each bar corresponds to a cup of a specific color, indicated by the bar's color. The Y-axis shows the percentage decrease in the object detection model's confidence score when a patch is projected onto the cup, compared to the confidence score without the patch.
  • ...and 4 more figures