Transient Adversarial 3D Projection Attacks on Object Detection in Autonomous Driving

TL;DR

The paper addresses the vulnerability of camera-based object detectors in autonomous driving to transient, physically realizable 3D projection attacks. It proposes a joint optimization framework that combines a color projection model and a TPS-based geometric transformation to synthesize a transient adversarial patch projected onto curved 3D surfaces, with data augmentation through Expectation Over Transformation to ensure robustness. The approach is demonstrated in indoor experiments targeting YOLOv3 and Mask R-CNN using a 1/10 scale RC car, achieving up to 100% misdetection under low ambient light and varying viewing conditions, highlighting practical risks in real-world scenarios. The authors discuss feasibility, limitations under ambient light and moving targets, and potential defenses such as adversarial training and temporal consistency checks to mitigate such projection attacks in AV systems.

Abstract

Object detection is a crucial task in autonomous driving. While existing research has proposed various attacks on object detection, such as those using adversarial patches or stickers, the exploration of projection attacks on 3D surfaces remains largely unexplored. Compared to adversarial patches or stickers, which have fixed adversarial patterns, projection attacks allow for transient modifications to these patterns, enabling a more flexible attack. In this paper, we introduce an adversarial 3D projection attack specifically targeting object detection in autonomous driving scenarios. We frame the attack formulation as an optimization problem, utilizing a combination of color mapping and geometric transformation models. Our results demonstrate the effectiveness of the proposed attack in deceiving YOLOv3 and Mask R-CNN in physical settings. Evaluations conducted in an indoor environment show an attack success rate of up to 100% under low ambient light conditions, highlighting the potential damage of our attack in real-world driving scenarios.

Figures (7)

• Figure 1: The attack scenarios of 3D projection attacks: ❶ The blue car and the white car are driving relatively static. ❷ The green car parks at the roadside. ❸ The traffic cone is placed at the entrance of the road construction area. The attackers position themselves either on the roadside or in another car. From these vantage points, the attacker can project an adversarial patch (projection light beams in yellow) onto the target vehicles or traffic cones, to render it undetectable by the victim AV (vehicles in yellow). This strategy could potentially lead to a collision between the victim AV and the target vehicles, or lead the victim vehicle into a dangerous road construction area.
• Figure 2: Overview of the adversarial patch generation pipeline.
• Figure 3: Examples of two viewing angles (b)(c) of the same (a) color board patch projection on the same vehicle.
• Figure 4: Attack setup. We use the projector to project the simulated adversarial patch on the target vehicle. The phone slider is used to collect videos when the victim's camera is moving.
• Figure 5: A visualization example of geometric transformation using TPS. (a) The color board used to collect TPS source control points; (b) The simulated color board after TPS transformation; (c) The simulated color board projected on the vehicle; (d) The projection of the color board on the real vehicle to collect TPS target control points; (e) The trained adversarial patch; (e) The simulated adversarial patch after geometric transformation; (g) The benign vehicle image; (h) The simulated patch projected on the vehicle.