Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD

Yijun Wan, Melih Barsbey, Abdellatif Zaidi, Umut Simsekli

TL;DR

This work studies provable neural network compressibility by perturbing SGD with heavy-tailed noise. By analyzing a mean-field two-layer network with the second layer fixed, it shows that injecting $\alpha$-stable noise yields a high-probability compression bound: for any compression ratio $\kappa$ there exists $N$ so that, for $n\ge N$ and $\eta\le n^{-\alpha/2-1}$, the relative pruning error satisfies $\frac{\|\hat{\Theta}^{(\kappa n)}_{\lfloor t/\eta \rfloor}-\hat{\Theta}^n_{\lfloor t/\eta \rfloor}\|_F}{\|\hat{\Theta}^n_{\lfloor t/\eta \rfloor}\|_F} \le \epsilon$ with high probability. The authors develop a rigorous pathway: SGD with heavy-tailed perturbations converges to a McKean–Vlasov SDE, a propagation of chaos result holds with rate $O(n^{-1/2})$, and strong discretization error bounds enable a practical Euler scheme; compressibility follows from heavy-tail packing arguments. Empirically, the approach improves compressibility and pruning robustness across ECG5000, MNIST, CIFAR10/100, with minimal degradation in pruned accuracy and even improved robustness on more complex architectures. The findings illuminate how carefully injected heavy-tailed noise can induce implicit compressibility in overparameterized models, with practical implications for model deployment and federated learning.

Abstract

Neural network compression has been an increasingly important subject, not only due to its practical relevance, but also due to its theoretical implications, as there is an explicit connection between compressibility and generalization error. Recent studies have shown that the choice of the hyperparameters of stochastic gradient descent (SGD) can have an effect on the compressibility of the learned parameter vector. These results, however, rely on unverifiable assumptions and the resulting theory does not provide a practical guideline due to its implicitness. In this study, we propose a simple modification for SGD, such that the outputs of the algorithm will be provably compressible without making any nontrivial assumptions. We consider a one-hidden-layer neural network trained with SGD, and show that if we inject additive heavy-tailed noise to the iterates at each iteration, for any compression rate, there exists a level of overparametrization such that the output of the algorithm will be compressible with high probability. To achieve this result, we make two main technical contributions: (i) we prove a 'propagation of chaos' result for a class of heavy-tailed stochastic differential equations, and (ii) we derive error estimates for their Euler discretization. Our experiments suggest that the proposed approach not only achieves increased compressibility with various models and datasets, but also leads to robust test performance under pruning, even in more realistic architectures that lie beyond our theoretical setting.

Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD

TL;DR

This work studies provable neural network compressibility by perturbing SGD with heavy-tailed noise. By analyzing a mean-field two-layer network with the second layer fixed, it shows that injecting -stable noise yields a high-probability compression bound: for any compression ratio there exists so that, for and , the relative pruning error satisfies with high probability. The authors develop a rigorous pathway: SGD with heavy-tailed perturbations converges to a McKean–Vlasov SDE, a propagation of chaos result holds with rate , and strong discretization error bounds enable a practical Euler scheme; compressibility follows from heavy-tail packing arguments. Empirically, the approach improves compressibility and pruning robustness across ECG5000, MNIST, CIFAR10/100, with minimal degradation in pruned accuracy and even improved robustness on more complex architectures. The findings illuminate how carefully injected heavy-tailed noise can induce implicit compressibility in overparameterized models, with practical implications for model deployment and federated learning.

Abstract

Neural network compression has been an increasingly important subject, not only due to its practical relevance, but also due to its theoretical implications, as there is an explicit connection between compressibility and generalization error. Recent studies have shown that the choice of the hyperparameters of stochastic gradient descent (SGD) can have an effect on the compressibility of the learned parameter vector. These results, however, rely on unverifiable assumptions and the resulting theory does not provide a practical guideline due to its implicitness. In this study, we propose a simple modification for SGD, such that the outputs of the algorithm will be provably compressible without making any nontrivial assumptions. We consider a one-hidden-layer neural network trained with SGD, and show that if we inject additive heavy-tailed noise to the iterates at each iteration, for any compression rate, there exists a level of overparametrization such that the output of the algorithm will be compressible with high probability. To achieve this result, we make two main technical contributions: (i) we prove a 'propagation of chaos' result for a class of heavy-tailed stochastic differential equations, and (ii) we derive error estimates for their Euler discretization. Our experiments suggest that the proposed approach not only achieves increased compressibility with various models and datasets, but also leads to robust test performance under pruning, even in more realistic architectures that lie beyond our theoretical setting.
Paper Structure (26 sections, 10 theorems, 64 equations, 2 figures, 7 tables)

This paper contains 26 sections, 10 theorems, 64 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Technical Background
  3. Problem Setting and the Main Result
  4. Proof Strategy and Intermediate Results
  5. Additional theoretical results.
  6. Empirical Results
  7. Experiments with ECG5000
  8. Experiments with MNIST
  9. Experiments with CIFAR10 and CIFAR100
  10. Robustness with More Complex Architectures
  11. Conclusion
  12. Incompressibility of the Brownian Case
  13. Local Convergence of SGD with $\alpha$-Stable Noise
  14. Technical Lemmas
  15. Propagation of Chaos
  16. ...and 11 more sections

Key Result

Theorem 3.1

Suppose that Assumption assump:coefficients_regularity holds. For any fixed $t>0$, $\kappa \in (0,1)$ and $\epsilon>0$ sufficiently small, with probability $1-\epsilon$, there exists $N\in\mathbb{N}_+$ such that for all $n\ge N$ and $\eta$ such that $\eta \le n^{-\alpha/2-1}$, the following upper bo

Figures (2)

  • Figure 1: The illustration of the overall approach. We consider a one-hidden-layer neural network with $n$ hidden units, which results in a weight matrix of $n$ columns (first layer). We show that, when SGD is perturbed with heavy-tailed noise, as $n\to \infty$, each column will follow a multivariate heavy-tailed distribution in an i.i.d. fashion. This implies that a small number of columns will have significantly larger norms compared to the others; hence, the norm of the overall weight matrix will be determined by such columns compressible_distribution. As a result, the majority can be removed (i.e., set to zero), which we refer to as compressibility.
  • Figure 2: Absolute and relative test accuracies after pruning, as a function of pruning ratio. Column \ref{['fig:cifar10_b']}: CIFAR10, FCN with a single hidden layer, $n=5$K. Column \ref{['fig:cifar10_cnn_b']}: CIFAR10, CNN. Column \ref{['fig:cifar100_b']}: CIFAR100, FCN with 4 hidden layers, $n=2048$.

Theorems & Definitions (21)

  • Theorem 3.1
  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Proposition A.1
  • proof
  • Proposition B.1
  • proof
  • Lemma C.1
  • proof
  • ...and 11 more