Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

George Philipp, Jaime G. Carbonell

TL;DR

The paper introduces the Nonlinearity Coefficient (NLC), a gradient-based metric that captures network nonlinearity and serves as a strong pre-training predictor of generalization across architectures. It formalizes the NLC and its theoretical ties to Jacobians and input/output covariances, and validates the approach with a large-scale empirical study spanning 750 random architectures on three datasets, identifying a narrow optimal NLC range (approximately 1–3) for generalization. The work demonstrates the NLC’s robustness to confounders like input scaling, bias, and width, and highlights the roles of output bias and skip connections in shaping performance. It also connects activation nonlinearity to NLC through activation-specific measures and discusses practical implications for architecture search, while outlining future extensions to CNNs and broader questions such as robustness and efficiency.

Abstract

For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding gradients, though even this guideline has remained relatively vague and circumstantial. We introduce the nonlinearity coefficient (NLC), a measurement of the complexity of the function computed by a neural network that is based on the magnitude of the gradient. Via an extensive empirical study, we show that the NLC is a powerful predictor of test error and that attaining a right-sized NLC is essential for optimal performance. The NLC exhibits a range of intriguing and important properties. It is closely tied to the amount of information gained from computing a single network gradient. It is tied to the error incurred when replacing the nonlinearity operations in the network with linear operations. It is not susceptible to the confounders of multiplicative scaling, additive bias and layer width. It is stable from layer to layer. Hence, we argue that the NLC is the first robust predictor of overfitting in deep networks.

The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

TL;DR

The paper introduces the Nonlinearity Coefficient (NLC), a gradient-based metric that captures network nonlinearity and serves as a strong pre-training predictor of generalization across architectures. It formalizes the NLC and its theoretical ties to Jacobians and input/output covariances, and validates the approach with a large-scale empirical study spanning 750 random architectures on three datasets, identifying a narrow optimal NLC range (approximately 1–3) for generalization. The work demonstrates the NLC’s robustness to confounders like input scaling, bias, and width, and highlights the roles of output bias and skip connections in shaping performance. It also connects activation nonlinearity to NLC through activation-specific measures and discusses practical implications for architecture search, while outlining future extensions to CNNs and broader questions such as robustness and efficiency.

Abstract

For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding gradients, though even this guideline has remained relatively vague and circumstantial. We introduce the nonlinearity coefficient (NLC), a measurement of the complexity of the function computed by a neural network that is based on the magnitude of the gradient. Via an extensive empirical study, we show that the NLC is a powerful predictor of test error and that attaining a right-sized NLC is essential for optimal performance. The NLC exhibits a range of intriguing and important properties. It is closely tied to the amount of information gained from computing a single network gradient. It is tied to the error incurred when replacing the nonlinearity operations in the network with linear operations. It is not susceptible to the confounders of multiplicative scaling, additive bias and layer width. It is stable from layer to layer. Hence, we argue that the NLC is the first robust predictor of overfitting in deep networks.

Paper Structure

This paper contains 47 sections, 1 equation, 12 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Notation and Terminology
  3. On the nonlinearity of neural networks - deriving the NLC
  4. Defining nonlinearity
  5. Definition 1.
  6. Defining the NLC
  7. Definition 2.
  8. On the predictive power of the NLC - large-scale empirical study
  9. Architectures used
  10. Datasets used
  11. Training protocol
  12. Presentation of results
  13. The NLC measures nonlinearity
  14. The NLC predicts test error
  15. NLC after training
  16. ...and 32 more sections

Figures (12)

  • Figure 1: NLC vs the median of the nonlinearity distribution. See section \ref{['derivationSection']} and \ref{['sensiDetails']} for details.
  • Figure 2: NLC versus test error. Points shown in red correspond to architectures with high output bias ($\sqrt{\mathbb{E}_x||f||_2^2} > 1000\sqrt{\mathbb{E}_x||f-\bar{f}||_2^2}$). Points shown in blue correspond to architectures that have skip connections. Inset graphs in the bottom right are magnifications of the region $0.8 < NLC < 100$. See section \ref{['nlcTestDetails']} for details.
  • Figure 3: Detailed results from our empirical study. See main text for explanation and sections \ref{['nlcTestDetails']} (figures A/B/C/D/F) and \ref{['errorRobustnessDetails']} (figure E) for further details.
  • Figure 4: The impact of confounders on test error (left y-axis) and various metrics (right y-axis). waveform-noise is the dataset. See section \ref{['confDetails']} for details.
  • Figure 5: Sawtooth activation function.
  • ...and 7 more figures