Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network

Tomer Galanti, Zachary S. Siegel, Aparna Gupte, Tomaso Poggio

TL;DR

It is demonstrated that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices, and it is predicted and empirically confirmed that weight decay is essential for this bias to occur.

Abstract

We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.

SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network

TL;DR

It is demonstrated that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices, and it is predicted and empirically confirmed that weight decay is essential for this bias to occur.

Abstract

We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.
Paper Structure (11 sections, 6 theorems, 21 equations, 17 figures, 1 table)

This paper contains 11 sections, 6 theorems, 21 equations, 17 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Theoretical Results
  4. Experiments
  5. Setup
  6. Results
  7. Conclusions
  8. Proofs
  9. Additional Experiments
  10. Experimental Details
  11. Results

Key Result

Lemma 3.0

Let $\ell$ be a differentiable loss function, and let $f_W$ be a model as described in Sec. sec:setup. For any weight matrix $W^l$ in $f_W$ and any sample $x \in \mathbb{R}^d$, the following inequality holds: where $m_l$ is a constant depending on the structure of the layer $l$ (defined in Eq. eq:def_f).

Figures (17)

  • Figure 1: Higher weight decay ($\lambda$) and learning rate ($\mu$), or smaller batch sizes ($B$), lead to a lower average rank across the network layers. We plot the average rank at end of training for ResNet-18 trained on CIFAR10 when varying a pair of hyperparameters.
  • Figure 2: Average ranks and accuracy rates of ResNet-18 trained on CIFAR10 when varying $\mu$. The top row shows the average rank across layers, while the bottom row shows the train and test accuracy rates for each setting. In this experiment, $\lambda={5}\mathrm{e}{-4}$ and $\epsilon={1}\mathrm{e}{-3}$.
  • Figure 3: Average ranks and accuracy rates of ResNet-18 trained on CIFAR10 when varying $\lambda$. In this experiment, $\mu=1.5$ and $\epsilon={1}\mathrm{e}{-3}$.
  • Figure 4: Average ranks and accuracy rates of ResNet-18 trained on CIFAR10 when varying $B$. In (a) we used $\mu={1}\mathrm{e}{-3}$ and $\lambda={6}\mathrm{e}{-3}$, in (b) we used $\mu={5}\mathrm{e}{-3}$ and $\lambda={6}\mathrm{e}{-3}$, and in (c) we used $\mu={1}\mathrm{e}{-2}$ and $\lambda={4}\mathrm{e}{-4}$. We used a threshold of $\epsilon={1}\mathrm{e}{-3}$.
  • Figure 5: Average rank of MLP-BN-10-100 trained on CIFAR10 when varying $\lambda$. In this experiment, $\mu=0.1$, momentum $0.9$ and $\epsilon={1}\mathrm{e}{-3}$.
  • ...and 12 more figures

Theorems & Definitions (9)

  • Lemma 3.0
  • Lemma 3.0
  • Theorem 3.1
  • Lemma A.0
  • proof
  • Lemma A.0
  • proof
  • Theorem A.1
  • proof