PAD: Patch-Agnostic Defense against Adversarial Patch Attacks

TL;DR

PAD tackles adversarial patch attacks on object detectors by identifying patches through two universal properties: semantic independence and spatial heterogeneity. It uses a patch localization pipeline based on a mutual-information heat map and a recompression-based heat map, fused in a Fusion Block, and then removes patches with simple inpainting, without requiring prior attack data or extra training. The method is compatible with any pre-trained detector and shows strong improvements across digital and physical patches, while maintaining near-unchanged performance on clean inputs. This patch-agnostic approach offers practical robustness and deployability for real-world detection systems.

Abstract

Adversarial patch attacks present a significant threat to real-world object detectors due to their practical feasibility. Existing defense methods, which rely on attack data or prior knowledge, struggle to effectively address a wide range of adversarial patches. In this paper, we show two inherent characteristics of adversarial patches, semantic independence and spatial heterogeneity, independent of their appearance, shape, size, quantity, and location. Semantic independence indicates that adversarial patches operate autonomously within their semantic context, while spatial heterogeneity manifests as distinct image quality of the patch area that differs from original clean image due to the independent generation process. Based on these observations, we propose PAD, a novel adversarial patch localization and removal method that does not require prior knowledge or additional training. PAD offers patch-agnostic defense against various adversarial patches, compatible with any pre-trained object detectors. Our comprehensive digital and physical experiments involving diverse patch types, such as localized noise, printable, and naturalistic patches, exhibit notable improvements over state-of-the-art works. Our code is available at https://github.com/Lihua-Jing/PAD.

Figures (6)

• Figure 1: When attacked by natural-looking adversarial patches, (a) locates high-frequency areas, and eliminates edge lines instead of patches naseer2019local; (b) fails to detect the existence of patches since no such patch data in the training set liu2022segment; (c) produces a heat map where the patch area and background are difficult to distinguish, failing to locate the patches tarchoun2023jedi. Our proposed PAD achieves accurate patch location and removal.
• Figure 2: Overview of our proposed PAD. Semantic Independence Localization and Spatial Heterogeneity Localization find patch regions from two views, generating heat maps feeding into Fusion Block. The patch localization map output by Fusion Block is then matched with all masks from SAM, getting more accurate patch boundaries. Feeding the defended image into object detectors for robust prediction.
• Figure 3: Visualization examples illustrating the patch localization process of PAD across different adversarial patch types.
• Figure 4: ASR (%) after defenses, lower values indicate better defense performance. Results with * are from liu2022segment.
• Figure 5: Comparison of defended images on APRICOT. Jedi tarchoun2023jedi fails to locate patches, the drop in ASR is caused by resizing in Auto-Encoder. SAC liu2022segment masks out most patch regions, but still affects many other regions despite the training with accurate mask annotations. PAD achieves the best removal performance.