Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum

TL;DR

The paper addresses the challenge of learning robust visual representations with self-supervised learning when pretraining data are heavily noisy. It introduces a denoiser-free framework by employing a noise curriculum (NC) that first initializes from denoised data and then adapts to noisier inputs, and a denoised-regularization variant (NCT) that anchors noisy embeddings to denoised ones via a frozen teacher. Empirical results on ImageNet-1k with extreme Gaussian noise show significant gains over standard DINOv2, with improvements up to $4.8\%$ in linear probing and notable gains under high-noise conditions, while preserving competitive performance on clean data. The approach also generalizes to other SSL models, reduces deployment complexity by removing the need for a denoiser at inference, and opens avenues for robust SSL across modalities and domains.

Abstract

Self-Supervised Learning (SSL) has become a powerful solution to extract rich representations from unlabeled data. Yet, SSL research is mostly focused on clean, curated and high-quality datasets. As a result, applying SSL on noisy data remains a challenge, despite being crucial to applications such as astrophysics, medical imaging, geophysics or finance. In this work, we present a fully self-supervised framework that enables noise-robust representation learning without requiring a denoiser at inference or downstream fine-tuning. Our method first trains an SSL denoiser on noisy data, then uses it to construct a denoised-to-noisy data curriculum (i.e., training first on denoised, then noisy samples) for pretraining a SSL backbone (e.g., DINOv2), combined with a teacher-guided regularization that anchors noisy embeddings to their denoised counterparts. This process encourages the model to internalize noise robustness. Notably, the denoiser can be discarded after pretraining, simplifying deployment. On ImageNet-1k with ViT-B under extreme Gaussian noise ($σ=255$, SNR = 0.72 dB), our method improves linear probing accuracy by 4.8% over DINOv2, demonstrating that denoiser-free robustness can emerge from noise-aware pretraining. The code is available at https://github.com/wenquanlu/noisy_dinov2.

Figures (23)

• Figure 1: Comparison of downstream pipelines with (left) and without (right) denoisers. The denoiser-free pipeline shows numerous advantages in efficiency, simplicity and robustness.
• Figure 2: Linear probing accuracies and noise examples on ImageNet-100 dataset. The denoiser-preprocessed baseline N2N + DINOv2 and our denoiser-free methods DINOv2 w/ NC and NCT significantly improve over the DINOv2 baseline. Note that DINOv2 w/ NC and N2N + DINOv2 overlap at moderate noise levels, and DINOv2 w/ NCT is only evaluated at Gaussian $\sigma = 255$.
• Figure 3: Overview of DINOv2 w/ NC which comprises SSL denoising, SSL representation learning and downstream tasks. A SSL denoiser is trained to denoise the data lowdosect. Then, DINOv2 is trained on the denoised data for early epochs followed by restarting training on noisy data. Lastly, after some fine-tuning, downstream tasks are performed directly on noisy inputs with an added prediction head.
• Figure 4: Structure of the denoised regularization loss. In addition to the original DINOv2 loss $L_{\mathrm{dinov2}}$, a regularization loss term $L_{\mathrm{dino\&ibot}}$ is introduced by comparing the output scores between the student and the frozen teacher. The frozen teacher takes in denoised inputs that undergo identical augmentations $\tau_t$ as the noisy inputs of the trainable teacher.
• Figure 5: Linear probing classification accuracies on DINOv2 backbone weights saved every 5 epochs during training on ImageNet-100 under various noise levels.