DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time

TL;DR

DynamicPAE addresses the challenge of generating scene-aware, real-time physical adversarial examples by learning a mapping from attacker observations to PAEs. It introduces residual-guided adversarial pattern exploration and distribution-matched attack scenario alignment to overcome noisy feedback and world-model misalignment. Empirical results demonstrate strong attack performance (2.07× average AP drop; 58.8% on average) and real-time inference (approximately 12 ms), across digital and physical setups, enabling end-to-end dynamic PAE modeling. The work advances the state of dynamic adversarial testing, offering a scalable framework and insights for defense and robust evaluation in real-world vision systems.

Abstract

Physical adversarial examples (PAEs) are regarded as whistle-blowers of real-world risks in deep-learning applications, thus worth further investigation. However, current PAE generation studies show limited adaptive attacking ability to diverse and varying scenes, revealing the urgent requirement of dynamic PAEs that are generated in real time and conditioned on the observation from the attacker. The key challenge in generating dynamic PAEs is learning the sparse relation between PAEs and the observation of attackers under the noisy feedback of attack training. To address the challenge, we present DynamicPAE, the first generative framework that enables scene-aware real-time physical attacks. Specifically, to address the noisy feedback problem that obfuscates the exploration of scene-related PAEs, we introduce the residual-guided adversarial pattern exploration technique. Residual-guided training, which relaxes the attack training with a reconstruction task, is proposed to enrich the feedback information, thereby achieving a more comprehensive exploration of PAEs. To address the alignment problem between the trained generator and the real-world scenario, we introduce the distribution-matched attack scenario alignment, consisting of the conditional-uncertainty-aligned data module and the skewness-aligned objective re-weighting module. The former aligns the training environment with the incomplete observation of the real-world attacker. The latter facilitates consistent stealth control across different attack targets with the skewness controller. Extensive digital and physical evaluations demonstrate the superior attack performance of DynamicPAE, attaining a 2.07 $\times$ boost (58.8% average AP drop under attack) on representative object detectors (e.g., DETR) over state-of-the-art static PAE generating methods. Overall, our work opens the door to end-to-end modeling of dynamic PAEs.

Figures (16)

• Figure 1: DynamicPAE framework trains real-time scene-aware physical adversarial example (PAE) generators and establishes a novel form of PAE, i.e., dynamic PAE. After training, the generator generates highly effective PAEs based on the characteristics of the current scene.
• Figure 2: The overview of the DynamicPAE framework, consisting of Residual Guided Adversarial Pattern Exploration that solves the training difficulties and the design methodology of Distribution-Matched Attack Scenario Alignment that aligns the generator to real-world constraints. The former enriches the guidance information with a residual-constructed reconstruction task, which decouples and diversifies the exploration of scene-conditional PAEs (shown as green & blue arrows). The latter adapts a probabilistic simulation process to model the real-world uncertainty, and a skewness-aligned objective weight controller for automatically balancing the stealthiness and attack objectives.
• Figure 3: Training Problem and Solution Illustration.
• Figure 4: Conditional Probabilistic Model Generating Dynamic Physical Adversarial Attack Scenes
• Figure 5: Skewness-based Hyperparameter Control

Theorems & Definitions (3)

• Definition 2.1
• Definition 3.1: Neural PAE Generator
• Proposition 3.1