On the Importance of Backbone to the Adversarial Robustness of Object Detectors

TL;DR

The paper addresses the vulnerability of object detectors to adversarial perturbations and demonstrates that prior robustness claims are inflated due to lacking adversarially pre-trained backbones. It introduces a simple yet effective recipe for fast adversarial fine-tuning that leverages upstream adversarial pre-training without modifying detector architectures, achieving state-of-the-art robustness across multiple detectors and tasks. Through thorough re-evaluation on VOC and COCO and analysis across detectors and backbones, the work shows backbone robustness dominates adversarial performance, with downstream modules playing a lesser role under the proposed regime. The findings generalize to panoptic segmentation, highlighting practical implications for building robust dense-prediction systems and guiding future pre-training and fine-tuning strategies.

Abstract

Object detection is a critical component of various security-sensitive applications, such as autonomous driving and video surveillance. However, existing object detectors are vulnerable to adversarial attacks, which poses a significant challenge to their reliability and security. Through experiments, first, we found that existing works on improving the adversarial robustness of object detectors give a false sense of security. Second, we found that adversarially pre-trained backbone networks were essential for enhancing the adversarial robustness of object detectors. We then proposed a simple yet effective recipe for fast adversarial fine-tuning on object detectors with adversarially pre-trained backbones. Without any modifications to the structure of object detectors, our recipe achieved significantly better adversarial robustness than previous works. Finally, we explored the potential of different modern object detector designs for improving adversarial robustness with our recipe and demonstrated interesting findings, which inspired us to design state-of-the-art (SOTA) robust detectors. Our empirical results set a new milestone for adversarially robust object detection. Code and trained checkpoints are available at https://github.com/thu-ml/oddefense.

Figures (8)

• Figure 1: Comparison between different training paradigms. The orange color indicates adversarially trained modules. (a) The standard training paradigm of object detectors. (b) The previous adversarial training paradigm on object detectors: Benignly pre-training the backbone on the upstream dataset and then adversarially training on the downstream detection dataset. (c) Adversarially pre-training the backbone on the upstream dataset and then adversarially training on the downstream dataset.
• Figure 2: Visualization of the detection results on benign images (upper) and $A_{\rm{cls}}$ adversarial images (lower), with three training methods STD (left), VANAT with the recipe of previous work (medium), and VANAT with our recipe (right). Faster R-CNN models were used as the detector.
• Figure 3: Evaluation results of detectors in various epoch settings on PASCAL VOC. (a) AP$_{50}$ on benign images. (b) AP$_{50}$ under $A_{\rm{cls}}$. Here the models were initialized by downstream benignly pre-trained backbones except for the red dashed line, which denotes the performance of the model trained by our recipe (24 epochs). The training cost is proportional to the epochs.
• Figure 4: Breakdown of errors on benign examples (upper) and $A_{\rm{cls}}$ adversarial examples (lower). Each curve is obtained by gradually relaxing the evaluation criteria. The severity of a particular error is reflected by the area between the curves, which is indicated in the legend. The errors are categorized as follows: C75: PR curve at IoU of 0.75, corresponding to AP$_{50}$. C50: PR curve at IoU of 0.75, corresponding to AP$_{75}$. Loc: false positives (FP) caused by poor localization. Sim: FP caused by confusion with similar objects. Oth: FP caused by confusion with other objects. BG: FP caused by confusion with background or unlabeled objects. FN: false negatives.
• Figure 5: Black-box transferability across object detectors trained by VANAT. The adversarial examples generated on the source models (each column) were fed into the target models (each row). The values denote the AP$_{50}$ of the target models on these adversarial images. The figure is divided into two parts according to the backbone of the target model for better comparison.