Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization

Samuel Schapiro, Han Zhao

TL;DR

This work empirically and theoretically investigates Sharpness-Aware Minimization (SAM) for out-of-distribution generalization. It conducts a comprehensive comparison of eight SAM variants on zero-shot OOD generalization, finding that SAM improves over Adam by $4.76\%$ on average and that the strongest variants reach $8.01\%$ on average. Extending the study to Gradual Domain Adaptation (GDA), SAM still yields improvements over Adam ($0.82\%$ average; strongest variants at $1.52\%$), and the authors derive corresponding sharpness-based generalization bounds for both OOD and GDA. The results reveal a gap between the empirical benefits of SAM and the asymptotic theory, motivating future work toward tighter analyses that better capture SAM’s practical gains in OOD settings.

Abstract

Recently, sharpness-aware minimization (SAM) has emerged as a promising method to improve generalization by minimizing sharpness, which is known to correlate well with generalization ability. Since the original proposal of SAM, many variants of SAM have been proposed to improve its accuracy and efficiency, but comparisons have mainly been restricted to the i.i.d. setting. In this paper we study SAM for out-of-distribution (OOD) generalization. First, we perform a comprehensive comparison of eight SAM variants on zero-shot OOD generalization, finding that the original SAM outperforms the Adam baseline by $4.76\%$ and the strongest SAM variants outperform the Adam baseline by $8.01\%$ on average. We then provide an OOD generalization bound in terms of sharpness for this setting. Next, we extend our study of SAM to the related setting of gradual domain adaptation (GDA), another form of OOD generalization where intermediate domains are constructed between the source and target domains, and iterative self-training is done on intermediate domains, to improve the overall target domain error. In this setting, our experimental results demonstrate that the original SAM outperforms the baseline of Adam on each of the experimental datasets by $0.82\%$ on average and the strongest SAM variants outperform Adam by $1.52\%$ on average. We then provide a generalization bound for SAM in the GDA setting. Asymptotically, this generalization bound is no better than the one for self-training in the literature of GDA. This highlights a further disconnection between the theoretical justification for SAM versus its empirical performance, with recent work finding that low sharpness alone does not account for all of SAM's generalization benefits. For future work, we provide several potential avenues for obtaining a tighter analysis for SAM in the OOD setting.

Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization

TL;DR

This work empirically and theoretically investigates Sharpness-Aware Minimization (SAM) for out-of-distribution generalization. It conducts a comprehensive comparison of eight SAM variants on zero-shot OOD generalization, finding that SAM improves over Adam by on average and that the strongest variants reach on average. Extending the study to Gradual Domain Adaptation (GDA), SAM still yields improvements over Adam ( average; strongest variants at ), and the authors derive corresponding sharpness-based generalization bounds for both OOD and GDA. The results reveal a gap between the empirical benefits of SAM and the asymptotic theory, motivating future work toward tighter analyses that better capture SAM’s practical gains in OOD settings.

Abstract

Recently, sharpness-aware minimization (SAM) has emerged as a promising method to improve generalization by minimizing sharpness, which is known to correlate well with generalization ability. Since the original proposal of SAM, many variants of SAM have been proposed to improve its accuracy and efficiency, but comparisons have mainly been restricted to the i.i.d. setting. In this paper we study SAM for out-of-distribution (OOD) generalization. First, we perform a comprehensive comparison of eight SAM variants on zero-shot OOD generalization, finding that the original SAM outperforms the Adam baseline by and the strongest SAM variants outperform the Adam baseline by on average. We then provide an OOD generalization bound in terms of sharpness for this setting. Next, we extend our study of SAM to the related setting of gradual domain adaptation (GDA), another form of OOD generalization where intermediate domains are constructed between the source and target domains, and iterative self-training is done on intermediate domains, to improve the overall target domain error. In this setting, our experimental results demonstrate that the original SAM outperforms the baseline of Adam on each of the experimental datasets by on average and the strongest SAM variants outperform Adam by on average. We then provide a generalization bound for SAM in the GDA setting. Asymptotically, this generalization bound is no better than the one for self-training in the literature of GDA. This highlights a further disconnection between the theoretical justification for SAM versus its empirical performance, with recent work finding that low sharpness alone does not account for all of SAM's generalization benefits. For future work, we provide several potential avenues for obtaining a tighter analysis for SAM in the OOD setting.

Paper Structure

This paper contains 42 sections, 7 theorems, 33 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Classifier and Loss
  4. Sharpness-Aware Minimization
  5. Background on SAM Variants
  6. A Study of SAM for Zero-Shot OOD Generalization
  7. Zero-Shot OOD Generalization
  8. Experiments Comparing SAM Variants for Zero-Shot OOD Generalization
  9. Datasets
  10. Model Setup
  11. Discussion of Results
  12. Understanding the FisherSAM Performance Gains
  13. Understanding the FriendlySAM Performance Gains
  14. Theoretical Analysis of SAM for OOD Generalization
  15. A Study of SAM for Gradual Domain Adaptation
  16. ...and 27 more sections

Key Result

Lemma 1

Given an error function ${\cal E}_\mu(\theta) := \mathbb{E}_{(\mathbf{x}, y) \sim \mu}[\ell(y, \mathbf{x}, \theta)]$ with loss satisfying Assumption lipschitz_loss and any distributions $\mu, \nu$ on ${\cal Y} \times {\cal X}$, we have that

Figures (1)

  • Figure 1: A comparison of SAM with varying levels of perturbation strength $\rho \in \{0, 0.01, 0.02, 0.05, 0.1, 0.2\}$, with $\rho = 0$ corresponding to Adam, on GDA datasets with varying levels of intermediate domains $T$. SAM outperforms Adam on all datasets: by 1.03% on Covertype, by 0.96% on Portraits, by 0.75% on Rotated MNIST, and by 0.53% on Color MNIST.

Theorems & Definitions (15)

  • Definition 1: $\rho$-Robust Risk
  • Definition 2: $\rho$-Sharpness
  • Definition 3: Wasserstein Distance
  • Lemma 1: Sharpness-Aware Error Difference Over Shifted Domains
  • Lemma 2: Sharpness-Aware Generalization Bound
  • Theorem 1: Sharpness-Aware Domain Adaptation Error
  • Definition 4: Distribution Shifts
  • Definition 5: Discrepancy Measure
  • Definition 6: Complete Binary Trees and Tree Path
  • Definition 7: Sequential Rademacher Complexity
  • ...and 5 more