With Great Backbones Comes Great Adversarial Transferability

TL;DR

The paper addresses the vulnerability of self-supervised learning (SSL) backbones when used to tune downstream vision models under a grey-box threat model. It introduces proxy models and a backbone-only attack to systematically quantify adversarial transferability across over $20{,}000$ configurations of tuning meta-information, backbone, and dataset, showing that partial knowledge can yield near white-box attack effectiveness. A key finding is that access to the pre-trained backbone weights alone can be as or more informative for crafting transferable adversarial samples as having full meta-information, with backbone attacks often outperforming strong black-box methods and approaching white-box performance. These results highlight substantial security risks in sharing public backbones and underscore the need for safer backbone-sharing practices and mitigation strategies in practical deployments.

Abstract

Advances in self-supervised learning (SSL) for machine vision have improved representation robustness and model performance, giving rise to pre-trained backbones like \emph{ResNet} and \emph{ViT} models tuned with SSL methods such as \emph{SimCLR}. Due to the computational and data demands of pre-training, the utilization of such backbones becomes a strenuous necessity. However, employing these backbones may inherit vulnerabilities to adversarial attacks. While adversarial robustness has been studied under \emph{white-box} and \emph{black-box} settings, the robustness of models tuned on pre-trained backbones remains largely unexplored. Additionally, the role of tuning meta-information in mitigating exploitation risks is unclear. This work systematically evaluates the adversarial robustness of such models across $20,000$ combinations of tuning meta-information, including fine-tuning techniques, backbone families, datasets, and attack types. We propose using proxy models to transfer attacks, simulating varying levels of target knowledge by fine-tuning these proxies with diverse configurations. Our findings reveal that proxy-based attacks approach the effectiveness of \emph{white-box} methods, even with minimal tuning knowledge. We also introduce a naive "backbone attack," leveraging only the backbone to generate adversarial samples, which outperforms \emph{black-box} attacks and rivals \emph{white-box} methods, highlighting critical risks in model-sharing practices. Finally, our ablations reveal how increasing tuning meta-information impacts attack transferability, measuring each meta-information combination.

Figures (5)

• Figure 1: The figure depicts all of the settings used to evaluate adversarial vulnerabilities given different information of the target model construction. From left to right, we simulate exhaustive varying combinations of meta-information available about the target model during adversarial attack construction. All of the created proxy models are used separately to assess adversarial transferability.
• Figure 2: The figure depicts the impact of the unavailability, i.e. difference from the target model, with each possible meta-information combination on adversarial transferability during proxy attack construction and the backbone attack. The results show the average difference from the white-box in transferability using PGD with a higher budget (left) and the segmentation w.r.t. in the target training mode (right).
• Figure 3: The figure breaks down impact of the unavailability, i.e. difference from the target model, of each possible meta-information combination on the change in the final decision-making of the model. Higher JS divergence implies a bigger change in the final classification of the sample.
• Figure 4: The figure depicts the impact of the unavailability, i.e. difference from the target model, of each possible meta-information combination on adversarial transferability during proxy attack construction and the backbone attack. The results show the average transferability for PGD with a higher budget for targeted vs untargeted attacks (left) and the segmentation w.r.t. the target training dataset (right).
• Figure 5: The figure shows scenarios where adversaries either know all meta-information but lack the weights or have access to the backbone weights (SwaV ResNet-50) alone. Knowledge of only the backbone is highlighted as BackbonePGD.