Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agnostic Sharpness-Aware Minimization

Van-Anh Nguyen, Quyen Tran, Tuan Truong, Thanh-Toan Do, Dinh Phung, Trung Le

TL;DR

A novel approach that combines the principles of both SAM and MAML, Agnostic-SAM, which significantly improves generalization over baselines across a range of datasets and under challenging conditions such as noisy labels or data limitation.

Abstract

Sharpness-aware minimization (SAM) has been instrumental in improving deep neural network training by minimizing both the training loss and the sharpness of the loss landscape, leading the model into flatter minima that are associated with better generalization properties. In another aspect, Model-Agnostic Meta-Learning (MAML) is a framework designed to improve the adaptability of models. MAML optimizes a set of meta-models that are specifically tailored for quick adaptation to multiple tasks with minimal fine-tuning steps and can generalize well with limited data. In this work, we explore the connection between SAM and MAML in enhancing model generalization. We introduce Agnostic-SAM, a novel approach that combines the principles of both SAM and MAML. Agnostic-SAM adapts the core idea of SAM by optimizing the model toward wider local minima using training data, while concurrently maintaining low loss values on validation data. By doing so, it seeks flatter minima that are not only robust to small perturbations but also less vulnerable to data distributional shift problems. Our experimental results demonstrate that Agnostic-SAM significantly improves generalization over baselines across a range of datasets and under challenging conditions such as noisy labels or data limitation.

Agnostic Sharpness-Aware Minimization

TL;DR

A novel approach that combines the principles of both SAM and MAML, Agnostic-SAM, which significantly improves generalization over baselines across a range of datasets and under challenging conditions such as noisy labels or data limitation.

Abstract

Sharpness-aware minimization (SAM) has been instrumental in improving deep neural network training by minimizing both the training loss and the sharpness of the loss landscape, leading the model into flatter minima that are associated with better generalization properties. In another aspect, Model-Agnostic Meta-Learning (MAML) is a framework designed to improve the adaptability of models. MAML optimizes a set of meta-models that are specifically tailored for quick adaptation to multiple tasks with minimal fine-tuning steps and can generalize well with limited data. In this work, we explore the connection between SAM and MAML in enhancing model generalization. We introduce Agnostic-SAM, a novel approach that combines the principles of both SAM and MAML. Agnostic-SAM adapts the core idea of SAM by optimizing the model toward wider local minima using training data, while concurrently maintaining low loss values on validation data. By doing so, it seeks flatter minima that are not only robust to small perturbations but also less vulnerable to data distributional shift problems. Our experimental results demonstrate that Agnostic-SAM significantly improves generalization over baselines across a range of datasets and under challenging conditions such as noisy labels or data limitation.
Paper Structure (28 sections, 2 theorems, 35 equations, 4 figures, 10 tables, 1 algorithm)

This paper contains 28 sections, 2 theorems, 35 equations, 4 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Sharpness-Aware Minimization.
  4. Model-Agnostic Machine Learning.
  5. Proposed Framework
  6. Notions
  7. Problem Formulation
  8. Our Solution
  9. Practical Algorithm.
  10. Experiments
  11. Image Classification From Scratch
  12. ImageNet dataset
  13. CIFAR dataset
  14. Transfer learning
  15. Train With Noisy Label
  16. ...and 13 more sections

Key Result

Theorem 1

Denote $\mathcal{L}_{\mathcal{D}}\left(\theta\mid S\right):=\max_{\theta':\Vert\theta'-\theta\Vert_{2}\leq\rho}\mathcal{L}_{S}\left(\theta'\right)$. Under some mild condition similar to SAM foret2021sharpnessaware, with a probability greater than $1 - \delta$ (i.e., $\delta \in [0,1]$) over the choi where $L$ is the upper-bound of the loss function (i.e., $\ell\left(x,y;\theta\right)\leq L,\forall

Figures (4)

  • Figure 1: Cosine similarity of two gradients $\nabla_{\theta}\mathcal{L}_{B^{t}}\left(\theta_{l}\right)$ and $\nabla_{\theta}\mathcal{L}_{B^{v}}\left(\tilde{\theta}_{l}^{v}\right)$ (a) before updating model $cosine_b$, (b) after updating model $cosine_a$ and (c) the improvement of this score $change$
  • Figure 2: Experiments of various perturbation radius $\rho_1$ and $\rho_2$
  • Figure 3: Loss landscape of EffecientNet-B2 trained on Flower102 dataset with (left) SGD, (middle) SAM, and (right) Agnostic-SAM.
  • Figure 4: Loss landscape of ResNet32 trained (left) SGD, (middle) SAM, and (right) Agnostic-SAM on Cifar100 dataset.

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • proof
  • proof