Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

Antoni Kowalczuk, Jan Dubiński, Atiyeh Ashari Ghomi, Yi Sui, George Stein, Jiapeng Wu, Jesse C. Cresswell, Franziska Boenisch, Adam Dziedzic

TL;DR

This work addresses the robustness of self-supervised vision encoders beyond image classification by benchmarking embedding-space and downstream-task attacks across semantic segmentation and depth estimation. Using EmbedAttack and task-specific PGD variants, it evaluates DINO and DINOv2 encoders, with and without DeACL adversarial fine-tuning. The findings show that embedding-based attacks remain potent across tasks, and DeACL improves robustness mainly for embedding perturbations while struggles against downstream attacks, highlighting limited cross-task robustness in current defenses. The study underscores the need for multi-perturbation adversarial training and task-aware robustness strategies to make SSL foundation models reliably multi-task in real-world settings.

Abstract

Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at ${github.com/layer6ai-labs/ssl-robustness}$.

Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

TL;DR

This work addresses the robustness of self-supervised vision encoders beyond image classification by benchmarking embedding-space and downstream-task attacks across semantic segmentation and depth estimation. Using EmbedAttack and task-specific PGD variants, it evaluates DINO and DINOv2 encoders, with and without DeACL adversarial fine-tuning. The findings show that embedding-based attacks remain potent across tasks, and DeACL improves robustness mainly for embedding perturbations while struggles against downstream attacks, highlighting limited cross-task robustness in current defenses. The study underscores the need for multi-perturbation adversarial training and task-aware robustness strategies to make SSL foundation models reliably multi-task in real-world settings.

Abstract

Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at .
Paper Structure (16 sections, 4 equations, 2 figures, 3 tables)

This paper contains 16 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Attack and Defense Methods
  4. Embedding-level Attack
  5. Downstream Attacks
  6. DeACL Defense
  7. Empirical Evaluation
  8. Setup
  9. Results
  10. Discussion and Conclusions
  11. Societal Impact
  12. Extended Related Work
  13. Adversarial Robustness in SSL
  14. Hyperparameters
  15. Further Insights on Depth Estimation
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Adversarial Attacks for SSL. An SSL encoder is applied to downstream tasks through adaptors. Adversarial attacks can attack the downstream labels, or the embedding space directly.
  • Figure 2: Evolution of robustness on different downstream tasks during DeACL fine-tuning.\ref{['fig:classification']} presents classification accuracy on clean data for the PGD and EmbedAttack attacks. \ref{['fig:segmentation']} shows segmentation mIoU on clean data for the SegPGD and EmbedAttack attacks. In \ref{['fig:depth_estimation']} we present depth estiation RMSE on clean data for the DepthPGD and EmbedAttack attacks. Line colors indicate different datasets and line styles indicate different reported metrics.