Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

Xin-Chun Li, Jin-Lin Tang, Bo Zhang, Lan Li, De-Chuan Zhan

TL;DR

This study methodically explores the factors affecting the symmetry of DNN valleys, encompassing the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and the magnitude and direction of the noise for 1D visualization, showing that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry.

Abstract

Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

TL;DR

This study methodically explores the factors affecting the symmetry of DNN valleys, encompassing the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and the magnitude and direction of the noise for 1D visualization, showing that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry.

Abstract

Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.
Paper Structure (33 sections, 2 equations, 29 figures, 1 table, 1 algorithm)

This paper contains 33 sections, 2 equations, 29 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Basic Notations and Preliminaries
  4. Previous Studies: Exploring $\theta_f$ with Fixed $\epsilon$
  5. Our Study: Exploring $\epsilon$ and $\lambda$ with Fixed $\theta_f$
  6. Experimental Findings
  7. Factors that Affect $\epsilon$
  8. Factors that Affect $\theta_f$
  9. BN and Initialization
  10. Hyperparameters
  11. Theoretical Insights to Explain the Finding
  12. Applications to Model Fusion
  13. Explaining the Success of Model Soups
  14. Regularizing the Sign Change in Federated Learning
  15. Limitations and Future Works
  16. ...and 18 more sections

Figures (29)

  • Figure 1: The illustration of investigated factors that could affect the valley symmetry. The $\epsilon$ matters a lot.
  • Figure 2: The illustration of different visualization methods for 1D visualization. The norm-scaled noise unifies the magnitude of various noise without changing their directions.
  • Figure 3: The valleys under 7 common noise types. The second row shows the results of replacing the sign of noise with that of $\theta_f$, leading to asymmetric valleys. (VGG16 with BN on CIFAR10)
  • Figure 4: The impacts of manually constructed Gaussian noise with different levels of sign consistency.
  • Figure 5: The valley shape under 6 special noise types. (VGG16 with BN on CIFAR10)
  • ...and 24 more figures