Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model

Gabriel Clara, Sophie Langer, Johannes Schmidt-Hieber

TL;DR

This work provides a rigorous, non-asymptotic analysis of gradient descent with dropout in the linear regression model. By formulating two dropout variants and examining both iterative updates and Ruppert-Polyak averaging, it disentangles the interaction between dropout-induced noise and gradient dynamics, revealing a persistent variance component beyond the marginalized loss. The authors derive precise covariance limits and show that averaging can recover the marginalized-optimal behavior, while a simplified dropout variant converges to ordinary least squares under invertible designs. The results illuminate the nuanced relationship between dropout and ell2-regularization in a tractable setting and lay groundwork for extending to stochastic gradients and more complex models.

Abstract

We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator

Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model

TL;DR

This work provides a rigorous, non-asymptotic analysis of gradient descent with dropout in the linear regression model. By formulating two dropout variants and examining both iterative updates and Ruppert-Polyak averaging, it disentangles the interaction between dropout-induced noise and gradient dynamics, revealing a persistent variance component beyond the marginalized loss. The authors derive precise covariance limits and show that averaging can recover the marginalized-optimal behavior, while a simplified dropout variant converges to ordinary least squares under invertible designs. The results illuminate the nuanced relationship between dropout and ell2-regularization in a tractable setting and lay groundwork for extending to stochastic gradients and more complex models.

Abstract

We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator
Paper Structure (23 sections, 19 theorems, 159 equations, 2 figures)

This paper contains 23 sections, 19 theorems, 159 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Other Related Work
  3. Notation
  4. Gradient Descent and Dropout
  5. Analysis of Averaged Dropout
  6. Analysis of Iterative Dropout Schemes
  7. Convergence of Dropout
  8. Ruppert-Polyak Averaging
  9. Convergence of Simplified Dropout
  10. Discussion and Outlook
  11. Proofs for Section \ref{['sec::avg_drop']}
  12. Proofs for Section \ref{['sec::gd_results']}
  13. Proof of Lemma \ref{['lem::wt_exp_conv']}
  14. Proof of Theorem \ref{['thm::gm']}
  15. Proof of Lemma \ref{['lem::wt_var_rec']}
  16. ...and 8 more sections

Key Result

Lemma 1

Given Assumption ass::sec_wtbeta_all, $\lVert[\rVert]{I - \alpha p \mathbb{X}_p} \leq 1 - \alpha p (1 - p) \min_{i} \mathbb{X}_{ii} < 1$ and for any $k = 0, 1, \ldots$,

Figures (2)

  • Figure 1: Regular neurons (left) with all connections active. Sample of the same neurons with dropout (right). The dashed connections are ignored during the current iteration of training.
  • Figure 2: The linear regression model $y = \sum_{i = 1}^d \beta_i x_i$, viewed as a neural network without hidden layers.

Theorems & Definitions (19)

  • Lemma 1: Convergence of Expectation
  • Theorem 1
  • Corollary 1
  • Lemma 2: Second Moment - Recursive Formula
  • Theorem 2: Second Moment - Limit Formula
  • Lemma 3
  • Theorem 3: Sub-Optimality of Variance
  • Theorem 4
  • Lemma 4
  • Theorem 5
  • ...and 9 more