Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Issues with Neural Tangent Kernel Approach to Neural Networks

Haoran Liu, Anthony Tai, David J. Crandall, Chunfeng Huang

TL;DR

This work critically examines the practical validity of the NTK-based equivalence between trained neural networks and kernel regression. Through rigorous derivations of NTK variants and extensive simulations, the authors show that increasing network depth does not yield matching improvements in predictor performance between neural nets and NTK-based kernel predictors, challenging the claimed equivalence. They find that Gaussian-process kernels corresponding to untrained networks can perform similarly to NTKs, and that alternative kernels can rival NTK-based predictions, suggesting NTKs may not fully capture neural-network training dynamics. The results urge a reevaluation of NTK-based analyses for understanding real-world network training and advocate developing models that better reflect the training process and nonlazy regimes.

Abstract

Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.

Issues with Neural Tangent Kernel Approach to Neural Networks

TL;DR

This work critically examines the practical validity of the NTK-based equivalence between trained neural networks and kernel regression. Through rigorous derivations of NTK variants and extensive simulations, the authors show that increasing network depth does not yield matching improvements in predictor performance between neural nets and NTK-based kernel predictors, challenging the claimed equivalence. They find that Gaussian-process kernels corresponding to untrained networks can perform similarly to NTKs, and that alternative kernels can rival NTK-based predictions, suggesting NTKs may not fully capture neural-network training dynamics. The results urge a reevaluation of NTK-based analyses for understanding real-world network training and advocate developing models that better reflect the training process and nonlazy regimes.

Abstract

Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.
Paper Structure (25 sections, 22 equations, 3 figures)

This paper contains 25 sections, 22 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions
  4. Background knowledge and methods used
  5. Fully Connected Neural Networks
  6. Connection between Neural Networks and Gaussian Processes
  7. A Brief Introduction To Neural Tangent Kernels
  8. The Equivalence Theorem
  9. Derivation of NTK and other models
  10. Models (Detailed derivations can be found in the appendix)
  11. Experiment Setup and Results
  12. Generating Simulated Data
  13. Results
  14. Conclusion and Discussion
  15. Appendix
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Boxplot of RMSE over 50 trials for each model
  • Figure 2: Table of Mean RMSE for each model.
  • Figure 3: table of mean RMSE and SE of mean RMSE