Texture Re-scalable Universal Adversarial Perturbation

TL;DR

This paper tackles the limitations of fixed texture and scale in universal adversarial perturbations by introducing Texture Scale Constrained UAP (TSC-UAP). TSC-UAP learns a small local texture patch v and tiles it into a full-size perturbation δ = 𝒯(v, α) with a split ratio α, optimizing over v under a norm constraint to maximize an attack loss. Across multiple datasets and models, including CNNs and ViT, TSC-UAP consistently improves fooling ratios and cross-model/data transferability, while maintaining low computational overhead and enabling data-efficient training. The approach demonstrates that category-specific local textures and controlled texture scale are a practical, general enhancement to both data-dependent and data-free UAP methods with broad potential impacts and future theoretical insights.

Abstract

Universal adversarial perturbation (UAP), also known as image-agnostic perturbation, is a fixed perturbation map that can fool the classifier with high probabilities on arbitrary images, making it more practical for attacking deep models in the real world. Previous UAP methods generate a scale-fixed and texture-fixed perturbation map for all images, which ignores the multi-scale objects in images and usually results in a low fooling ratio. Since the widely used convolution neural networks tend to classify objects according to semantic information stored in local textures, it seems a reasonable and intuitive way to improve the UAP from the perspective of utilizing local contents effectively. In this work, we find that the fooling ratios significantly increase when we add a constraint to encourage a small-scale UAP map and repeat it vertically and horizontally to fill the whole image domain. To this end, we propose texture scale-constrained UAP (TSC-UAP), a simple yet effective UAP enhancement method that automatically generates UAPs with category-specific local textures that can fool deep models more easily. Through a low-cost operation that restricts the texture scale, TSC-UAP achieves a considerable improvement in the fooling ratio and attack transferability for both data-dependent and data-free UAP methods. Experiments conducted on two state-of-the-art UAP methods, eight popular CNN models and four classical datasets show the remarkable performance of TSC-UAP.

Figures (11)

• Figure 1: We propose TSC-UAP, an enhancement method that utilizes texture scale constraints to restrict the UAP. The texture scale constraints facilitate UAP to achieve higher fooling ratios, no matter data-dependent or data-free.
• Figure 2: (a): UAP generated by SGD-UAP. The UAP achieves 80.3% fooling ratio. (b) Attention map of (a). (c): Replace the center of (a) with textures in the top left corner of (a) (i.e., textures in the pink box). The UAP achieves 81.1% fooling ratio. (d): Replace the center of (a) with the replicated local textures in (a) (i.e., textures in the yellow box). The UAP achieves 84.3% fooling ratio.
• Figure 3: We choose the pipeline of SGD-UAP since the method is concise. It can represent both data-dependent and data-free UAP generation methods. In the standard UAP generation pipeline, they add UAP to the clean images and put the perturbed images into the classifier to calculate the loss. Our modification is in the green box with red bounds. The difference is that we use the patch (i.e., category-specific local texture) and texture scale constraints (i.e., split ratio $\alpha$) to obtain the UAP. The gradient back-propagation is applied on the patch, not the UAP.
• Figure 4: Here shows the human understandable UAPs generated by attacking ResNet50, VGG19, DenseNet121 and MobileNet-v2 on ImageNet training samples.
• Figure 5: Untargeted UAPs generated with different texture scale constraints (i.e., $\alpha$) on ImageNet training samples.