A Single Set of Adversarial Clothes Breaks Multiple Defense Methods in the Physical World

TL;DR

This work reveals a systemic vulnerability of state-of-the-art physical adversarial defenses by showing that adversarial clothes—textures applied to garments—can defeat multiple defense methods. By first enlarging patch size and then optimizing textures via a 3D rendering pipeline with transformation-robustness, the authors craft clothes that reduce detector performance across nine defenses in both digital and physical tests. They demonstrate that an ensemble of defended models can be bypassed, achieving high attack success rates and exposing fundamental weaknesses in current robustness approaches. The findings underscore the need for defense strategies that account for texture-based, real-world adversaries and broader threat models beyond patch-centric schemes.

Abstract

In recent years, adversarial attacks against deep learning-based object detectors in the physical world have attracted much attention. To defend against these attacks, researchers have proposed various defense methods against adversarial patches, a typical form of physically-realizable attack. However, our experiments showed that simply enlarging the patch size could make these defense methods fail. Motivated by this, we evaluated various defense methods against adversarial clothes which have large coverage over the human body. Adversarial clothes provide a good test case for adversarial defense against patch-based attacks because they not only have large sizes but also look more natural than a large patch on humans. Experiments show that all the defense methods had poor performance against adversarial clothes in both the digital world and the physical world. In addition, we crafted a single set of clothes that broke multiple defense methods on Faster R-CNN. The set achieved an Attack Success Rate (ASR) of 96.06% against the undefended detector and over 64.84% ASRs against nine defended models in the physical world, unveiling the common vulnerability of existing adversarial defense methods against adversarial clothes. Code is available at: https://github.com/weiz0823/adv-clothes-break-multiple-defenses.

Figures (8)

• Figure 1: APs of person class of the FNC-defended Faster R-CNN frcnn detector against patch attack of different patch sizes on the Inria person dataset inria. Solid lines denote the APs of the FNC-defended detector, when the input images are applied with random patch, non-adaptive patch optimized on the undefended detector, and adaptive patch optimized on the FNC-defended detector. Dashed line denotes the AP of the undefended detector, when the input images are applied with non-adaptive patch. Details of experiment setups are provided in \ref{['sec:expand-patch']}.
• Figure 2: Visualization of the patches of different sizes when applied onto persons in the Inria inria dataset. The images have been cropped and zoomed in for a better view.
• Figure 3: Illustration of position jittering crops to enhance the physical world robustness against texture-based attacks. The whole texture represents the latent feature map. The red boxes are possible texture crops used for optimization, and the yellow box is the center texture crop used for evaluation and physical implementation.
• Figure 4: APs of Faster R-CNN versus jittering scale in the digital world for different jittering scales during texture optimization. $\gamma=0$ denotes no position jittering.
• Figure 5: Visualization of different adversarial clothes produced in the physical world.