Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weight Conditioning for Smooth Optimization of Neural Networks

Hemanth Saratchandran, Thomas X. Wang, Simon Lucey

TL;DR

A novel normalization technique for neural network weight matrices, which is term weight conditioning, aims to narrow the gap between the smallest and largest singular values of the weight matrices, resulting in better-conditioned matrices.

Abstract

In this article, we introduce a novel normalization technique for neural network weight matrices, which we term weight conditioning. This approach aims to narrow the gap between the smallest and largest singular values of the weight matrices, resulting in better-conditioned matrices. The inspiration for this technique partially derives from numerical linear algebra, where well-conditioned matrices are known to facilitate stronger convergence results for iterative solvers. We provide a theoretical foundation demonstrating that our normalization technique smoothens the loss landscape, thereby enhancing convergence of stochastic gradient descent algorithms. Empirically, we validate our normalization across various neural network architectures, including Convolutional Neural Networks (CNNs), Vision Transformers (ViT), Neural Radiance Fields (NeRF), and 3D shape modeling. Our findings indicate that our normalization method is not only competitive but also outperforms existing weight normalization techniques from the literature.

Weight Conditioning for Smooth Optimization of Neural Networks

TL;DR

A novel normalization technique for neural network weight matrices, which is term weight conditioning, aims to narrow the gap between the smallest and largest singular values of the weight matrices, resulting in better-conditioned matrices.

Abstract

In this article, we introduce a novel normalization technique for neural network weight matrices, which we term weight conditioning. This approach aims to narrow the gap between the smallest and largest singular values of the weight matrices, resulting in better-conditioned matrices. The inspiration for this technique partially derives from numerical linear algebra, where well-conditioned matrices are known to facilitate stronger convergence results for iterative solvers. We provide a theoretical foundation demonstrating that our normalization technique smoothens the loss landscape, thereby enhancing convergence of stochastic gradient descent algorithms. Empirically, we validate our normalization across various neural network architectures, including Convolutional Neural Networks (CNNs), Vision Transformers (ViT), Neural Radiance Fields (NeRF), and 3D shape modeling. Our findings indicate that our normalization method is not only competitive but also outperforms existing weight normalization techniques from the literature.
Paper Structure (22 sections, 3 theorems, 18 equations, 7 figures, 4 tables)

This paper contains 22 sections, 3 theorems, 18 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Normalization in deep learning:
  4. Notation
  5. Motivation
  6. A simple model:
  7. Preconditioning:
  8. General objective functions:
  9. Theoretical Methodology
  10. Weight conditioning for feed forward networks
  11. Experiments
  12. Convolutional Neural Networks (CNNs)
  13. Experimental setup:
  14. Inception:
  15. DenseNet:
  16. ...and 7 more sections

Key Result

theorem thmcountertheorem

Let $A$ be a $n\times m$ matrix, $P$ an arbitrary diagonal $n \times n$ matrix and $E$ the row equilibrated matrix built from $A$. Then $\kappa(EA) \leq \kappa(PA)$.

Figures (7)

  • Figure 1: Left; Testing three different normalizations on the GoogleNet CNN trained on CIFAR100. BN + WC (ours) reaches a much higher accuracy than the other two. Right; Testing the same three normalizations on a ResNet18 and ResNet50 CNN architecture. In both cases BN + WC (ours) performs better.
  • Figure 2: Schematic representation of a preconditioned network. The weights from the neurons (red) are first multiplied by a preconditioner matrix (green) before being activated (orange).
  • Figure 3: Left; Train loss curves for four normalization schemes on an Inception architecture trained on the CIFAR10 dataset. Right; Top-1% accuracy plotted during training. We see that BN + E converges the fastest.
  • Figure 4: Left; Train loss curves for four normalization schemes on an Inception architecture trained on the CIFAR100 dataset. Right; Top-1% accuracy plotted during training. We see that BN + E converges the fastest.
  • Figure 5: Left; Train loss curves for four normalization schemes on a DenseNet architecture trained on CIFAR10. Right; Top-1% accuracy plotted during training. We see that BN + E yields higher Top-1% accuracy than the other three.
  • ...and 2 more figures

Theorems & Definitions (6)

  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • theorem thmcountertheorem: Van Der Sluis van1969condition
  • proposition thmcounterproposition
  • proof
  • theorem thmcountertheorem