Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simple Linear Neuron Boosting

Daniel Munoz

TL;DR

The paper introduces Linear Neuron Boosting (LNB), a function-space optimization technique that preconditions gradient updates for linear neurons by whitening their input features. By formulating the per-neuron metric $M_i = \mathbb{E}_{x_{i-1}\sim X_i}[ (\partial x_i / \partial \theta_i)^T (\partial x_i / \partial \theta_i) ]$ and solving $M_i \hat{\theta}_i = g_i$, LNB achieves a preconditioned, matrix-free update $\hat{\theta}_B$ that is equivalent to a whitening reparameterization and can be implemented with autodifferentiation across architectures. The method supports online learning via EMA estimates and limited Conjugate Gradient iterations, with a straightforward interpretation as feature whitening preceding linear transforms. Empirical results across matrix factorization, MLPs, Vision Transformers, and UNet demonstrate faster convergence in epochs and competitive wall-clock time relative to Adam, highlighting LNB as a practical, architecture-agnostic preconditioning technique. The work situates LNB among function-space optimizations and connects it to whitening and second-order methods, offering a scalable, easy-to-implement alternative for improving training dynamics in deep networks.

Abstract

Given a differentiable network architecture and loss function, we revisit optimizing the network's neurons in function space using Boosted Backpropagation (Grubb & Bagnell, 2010), in contrast to optimizing in parameter space. From this perspective, we reduce descent in the space of linear functions that optimizes the network's backpropagated-errors to a preconditioned gradient descent algorithm. We show that this preconditioned update rule is equivalent to reparameterizing the network to whiten each neuron's features, with the benefit that the normalization occurs outside of inference. In practice, we use this equivalence to construct an online estimator for approximating the preconditioner and we propose an online, matrix-free learning algorithm with adaptive step sizes. The algorithm is applicable whenever autodifferentiation is available, including convolutional networks and transformers, and it is simple to implement for both the local and distributed training settings. We demonstrate fast convergence both in terms of epochs and wall clock time on a variety of tasks and networks.

Simple Linear Neuron Boosting

TL;DR

The paper introduces Linear Neuron Boosting (LNB), a function-space optimization technique that preconditions gradient updates for linear neurons by whitening their input features. By formulating the per-neuron metric and solving , LNB achieves a preconditioned, matrix-free update that is equivalent to a whitening reparameterization and can be implemented with autodifferentiation across architectures. The method supports online learning via EMA estimates and limited Conjugate Gradient iterations, with a straightforward interpretation as feature whitening preceding linear transforms. Empirical results across matrix factorization, MLPs, Vision Transformers, and UNet demonstrate faster convergence in epochs and competitive wall-clock time relative to Adam, highlighting LNB as a practical, architecture-agnostic preconditioning technique. The work situates LNB among function-space optimizations and connects it to whitening and second-order methods, offering a scalable, easy-to-implement alternative for improving training dynamics in deep networks.

Abstract

Given a differentiable network architecture and loss function, we revisit optimizing the network's neurons in function space using Boosted Backpropagation (Grubb & Bagnell, 2010), in contrast to optimizing in parameter space. From this perspective, we reduce descent in the space of linear functions that optimizes the network's backpropagated-errors to a preconditioned gradient descent algorithm. We show that this preconditioned update rule is equivalent to reparameterizing the network to whiten each neuron's features, with the benefit that the normalization occurs outside of inference. In practice, we use this equivalence to construct an online estimator for approximating the preconditioner and we propose an online, matrix-free learning algorithm with adaptive step sizes. The algorithm is applicable whenever autodifferentiation is available, including convolutional networks and transformers, and it is simple to implement for both the local and distributed training settings. We demonstrate fast convergence both in terms of epochs and wall clock time on a variety of tasks and networks.

Paper Structure

This paper contains 22 sections, 15 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation and Background
  3. Preconditioned Gradient Descent
  4. Natural Gradient Descent
  5. Functional Gradient Descent
  6. Regularization
  7. Boosted Backpropagation
  8. Boosting over Linear Neurons
  9. Reduction
  10. Optimization
  11. Update Properties
  12. Positive Semi-definite Metrics
  13. Online Learning
  14. Complexity
  15. Related Work
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Matrix factorization via a 2-layer linear network
  • Figure 2: MNIST test accuracy evolution trained on the original data (a) vs. inverted pixels (b). The step-size is parenthesized.
  • Figure 3: ViT performances on CIFAR10.
  • Figure 4: UNet performances on VOC Segmentation.