Revisiting Physical-World Adversarial Attack on Traffic Sign Recognition: A Commercial Systems Perspective

TL;DR

The paper tackles the practicality of physical-world adversarial attacks on commercial traffic sign recognition (TSR) by conducting the first large-scale assessment across multiple real-world vehicle models. It reveals limited generalizability of academic attacks at the system level due to a spatial memorization design in commercial TSR, and introduces surrogate system-level metrics SysHA and SysAA to quantify this effect. The study shows overall transfer success is low (6.67%), with some niche configurations reaching 100% for specific signs, and it reevaluates prior attacks under the new metrics, yielding seven new observations. These findings have direct implications for evaluating TSR security in production systems and guide future defense-oriented research. The work also provides data and methodologies to better align TSR security research with real-world commercial deployments.

Abstract

Traffic Sign Recognition (TSR) is crucial for safe and correct driving automation. Recent works revealed a general vulnerability of TSR models to physical-world adversarial attacks, which can be low-cost, highly deployable, and capable of causing severe attack effects such as hiding a critical traffic sign or spoofing a fake one. However, so far existing works generally only considered evaluating the attack effects on academic TSR models, leaving the impacts of such attacks on real-world commercial TSR systems largely unclear. In this paper, we conduct the first large-scale measurement of physical-world adversarial attacks against commercial TSR systems. Our testing results reveal that it is possible for existing attack works from academia to have highly reliable (100\%) attack success against certain commercial TSR system functionality, but such attack capabilities are not generalizable, leading to much lower-than-expected attack success rates overall. We find that one potential major factor is a spatial memorization design that commonly exists in today's commercial TSR systems. We design new attack success metrics that can mathematically model the impacts of such design on the TSR system-level attack success, and use them to revisit existing attacks. Through these efforts, we uncover 7 novel observations, some of which directly challenge the observations or claims in prior works due to the introduction of the new metrics.

Figures (7)

• Figure 1: Experiment setup for commercial TSR system testing. We cover the vehicle in the photo for anonymity purpose.
• Figure 2: Visualisation of the hiding attacks (HA) generated for STOP and speed limit signs, which are used in our commercial TSR systems testing. They are generated by the three most promising prior works (RP$_2$eykholt2018physical, SIB zhao2019seeing, FTE jia2022fooling) using surrogate models of both representative one-stage and two-stage TSR model designs.
• Figure 3: Experimental setup for our investigation into the spatial memorization design in commercial TSR systems. As shown, we first show the sign to the vehicle for a short time (sign display time), and hide the sign and wait for a certain time duration (sign disappearing time). After that, we drive the vehicle past the original sign-display position to measure whether the sign detection result is spatially memorized.
• Figure 4: Illustration of the potentially misleading effect of existing TSR model-level metrics with respect to the TSR system-level attack success. As shown, although $f_\mathrm{HA}$ and $f_\mathrm{AA}$ are both 50% for this scenario, the TSR system-level attack success rates are in fact 0% for hiding attack (HA) and 100% for appearing attack (AA) due to spatial memorization.
• Figure 5: Setup for calculating the proposed surrogate TSR system-level attack success metric designs (SysHA and SysAA, detailed design in §\ref{['sec:new_metric']}).