Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detection

TL;DR

This work tackles the problem of black-box adversarial attacks against object detectors by introducing PRFA, a query-efficient method that operates without gradients. PRFA reduces search complexity with prior-guided dimensionality reduction, accelerates exploration through parallel breadth search, and enhances effectiveness via rectangle flips to perturb object-contour regions, yielding transferable adversarial examples across anchor-based and anchor-free detectors. The approach achieves stronger untargeted attacks with fewer queries than strong baselines and demonstrates cross-model transferability, underscoring practical security risks for safety-critical systems like autonomous driving. Overall, PRFA offers a principled, efficient framework for exploiting detection vulnerabilities in realistic black-box settings.

Abstract

Object detection has been widely used in many safety-critical tasks, such as autonomous driving. However, its vulnerability to adversarial examples has not been sufficiently studied, especially under the practical scenario of black-box attacks, where the attacker can only access the query feedback of predicted bounding-boxes and top-1 scores returned by the attacked model. Compared with black-box attack to image classification, there are two main challenges in black-box attack to detection. Firstly, even if one bounding-box is successfully attacked, another sub-optimal bounding-box may be detected near the attacked bounding-box. Secondly, there are multiple bounding-boxes, leading to very high attack cost. To address these challenges, we propose a Parallel Rectangle Flip Attack (PRFA) via random search. We explain the difference between our method with other attacks in Fig.~\ref{fig1}. Specifically, we generate perturbations in each rectangle patch to avoid sub-optimal detection near the attacked region. Besides, utilizing the observation that adversarial perturbations mainly locate around objects' contours and critical points under white-box attacks, the search space of attacked rectangles is reduced to improve the attack efficiency. Moreover, we develop a parallel mechanism of attacking multiple rectangles simultaneously to further accelerate the attack process. Extensive experiments demonstrate that our method can effectively and efficiently attack various popular object detectors, including anchor-based and anchor-free, and generate transferable adversarial examples.

Figures (5)

• Figure 1: We use the coordinate axis to show the taxonomy of the adversarial attack in object detection. Different from white box attacks and migration attacks, our method PRFA only relies on the prediction box and top-1 score output after NMS to attack through queries without gradients. PRFA can also attack anchor-free and anchor-based models at the same time.
• Figure 2: The adversarial perturbations generated by the white-box attack methods UEA and DAG respectively on the SSD and Faster-RCNN. These perturbations are basically distributed at the contours and critical points of the object.
• Figure 3: The process of flipping perturbation's sign. We force the points with the same feature to be different by flipping sign, which will cause the detector to separate them to achieve an effective attack.
• Figure 4: $mAP$ changes $w.r.t.$ the number of queries for different attacks. FPRA achieves the fastest convergence and the most effective attack.
• Figure 5: The detection results of the four detectors on clean images (yellow lines) and adversarial samples (red lines).