T2I-Based Physical-World Appearance Attack against Traffic Sign Recognition Systems in Autonomous Driving

TL;DR

This work tackles the vulnerability of traffic sign recognition systems to physical-world adversarial appearances by introducing DiffSign, a diffusion-model–based framework that yields stealthy, transferable adversarial signs. DiffSign uses cropping to focus on the primary object, CLIP-guided losses, masked prompts, and two style-customization strategies (image- and prompt-specified) to improve focus, controllability, and generalization to out-of-domain signs. Extensive real-world evaluations show DiffSign achieves an average physical-world attack success rate of about $83.3\%$, with up to $97\%$ transfer to commercial TSR systems and full end-to-end impact in simulations, while remaining largely inconspicuous to human observers. The results highlight both the susceptibility of TSR to advanced diffusion-based attacks and the need for defenses that incorporate strong textual priors and robust patch localization, informing future research on secure autonomous driving perception.

Abstract

Traffic Sign Recognition (TSR) systems play a critical role in Autonomous Driving (AD) systems, enabling real-time detection of road signs, such as STOP and speed limit signs. While these systems are increasingly integrated into commercial vehicles, recent research has exposed their vulnerability to physical-world adversarial appearance attacks. In such attacks, carefully crafted visual patterns are misinterpreted by TSR models as legitimate traffic signs, while remaining inconspicuous or benign to human observers. However, existing adversarial appearance attacks suffer from notable limitations. Pixel-level perturbation-based methods often lack stealthiness and tend to overfit to specific surrogate models, resulting in poor transferability to real-world TSR systems. On the other hand, text-to-image (T2I) diffusion model-based approaches demonstrate limited effectiveness and poor generalization to out-of-distribution sign types. In this paper, we present DiffSign, a novel T2I-based appearance attack framework designed to generate physically robust, highly effective, transferable, practical, and stealthy appearance attacks against TSR systems. To overcome the limitations of prior approaches, we propose a carefully designed attack pipeline that integrates CLIP-based loss and masked prompts to improve attack focus and controllability. We also propose two novel style customization methods to guide visual appearance and improve out-of-domain traffic sign attack generalization and attack stealthiness. We conduct extensive evaluations of DiffSign under varied real-world conditions, including different distances, angles, light conditions, and sign categories. Our method achieves an average physical-world attack success rate of 83.3%, leveraging DiffSign's high effectiveness in attack transferability.

Figures (12)

• Figure 1: Visualization of representative existing TSR attacks: SIB zhao2019seeing on STOP sign and FTE jia2022fooling on speed limit sign, compared with our newly proposed attack DiffSign, respectively.
• Figure 2: Example of our attack scenario in the real world (Left) and a schematic of the appearance attack (Right).
• Figure 3: Design overview of our attack, DiffSign, a T2I-based physical-world appearance attack.
• Figure 4: Motivation examples of our crop mechanism and BBOX filter design: (a) crop mechanism to focus optimization on primary objects and (b) the necessity of a BBOX filter.
• Figure 5: Visualization examples of image-specified style customization. Left is the specified style image. Right is generated adversarial appearance attack.