Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence guarantees for forward gradient descent in the linear regression model

Thijs Bos, Johannes Schmidt-Hieber

TL;DR

This work analyzes weight-perturbed forward gradient descent for the linear regression model with random design, motivated by biologically plausible learning. It shows that the forward-gradient updates are gradient-descent in expectation and derives a covariance-recursion-based bound on the error, enabling a finite-sample MSE rate. With a carefully chosen stepsize, the bound yields a rate of $d^2 \log(d)/k$ once the number of samples satisfies $k \gtrsim e^2 d^2 \log(d)$, revealing an extra $d \log(d)$ factor compared with gradient-based methods. The results illuminate the trade-offs of gradient-free learning in high dimensions and suggest potential improvements by reusing data points or extending the analysis beyond linear models.

Abstract

Renewed interest in the relationship between artificial and biological neural networks motivates the study of gradient-free methods. Considering the linear regression model with random design, we theoretically analyze in this work the biologically motivated (weight-perturbed) forward gradient scheme that is based on random linear combination of the gradient. If d denotes the number of parameters and k the number of samples, we prove that the mean squared error of this method converges for $k\gtrsim d^2\log(d)$ with rate $d^2\log(d)/k.$ Compared to the dimension dependence d for stochastic gradient descent, an additional factor $d\log(d)$ occurs.

Convergence guarantees for forward gradient descent in the linear regression model

TL;DR

This work analyzes weight-perturbed forward gradient descent for the linear regression model with random design, motivated by biologically plausible learning. It shows that the forward-gradient updates are gradient-descent in expectation and derives a covariance-recursion-based bound on the error, enabling a finite-sample MSE rate. With a carefully chosen stepsize, the bound yields a rate of once the number of samples satisfies , revealing an extra factor compared with gradient-based methods. The results illuminate the trade-offs of gradient-free learning in high dimensions and suggest potential improvements by reusing data points or extending the analysis beyond linear models.

Abstract

Renewed interest in the relationship between artificial and biological neural networks motivates the study of gradient-free methods. Considering the linear regression model with random design, we theoretically analyze in this work the biologically motivated (weight-perturbed) forward gradient scheme that is based on random linear combination of the gradient. If d denotes the number of parameters and k the number of samples, we prove that the mean squared error of this method converges for with rate Compared to the dimension dependence d for stochastic gradient descent, an additional factor occurs.
Paper Structure (6 sections, 4 theorems, 46 equations, 2 figures)

This paper contains 6 sections, 4 theorems, 46 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Weight-perturbed forward gradient descent
  3. Convergence rates in the linear regression model
  4. Proofs
  5. Proof of Theorem \ref{['T:Covariance']}
  6. Proof of Theorem \ref{['T:Convergence bound']}

Key Result

Theorem 3.1

We have $\mathbb{E}[\bm{\theta}_{k}]-\bm{\theta}_{\star}=(\mathbf{I}_d-\alpha_k\Sigma)(\mathbb{E}[\bm{\theta}_{k-1}]-\bm{\theta}_{\star})$ and thus

Figures (2)

  • Figure 1: Computional graphs for computing in a forward pass $L(\bm{\theta})=\frac{1}{2}(Y-X_1\theta_1-X_2\theta_2)^2$ (upper half) and $(\nabla L(\bm{\theta}))^\top\mathbf{v}$ (lower half).
  • Figure 2: Comparison of the MSE of forward gradient descent (blue) and SGD (red) for dimensions $d=10$ and $d=100.$ The upper dashed line is $k\mapsto d^2\log(d)/k$, the middle dashed line is $k\mapsto d^2/k$, and the lower dashed line is $k\mapsto d/k.$

Theorems & Definitions (7)

  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3: Mean squared error
  • proof : Proof of Theorem \ref{['T: Expectations']}
  • Lemma 4.1
  • proof
  • proof : Proof of Theorem \ref{['T:Covariance']}