Correspondence of NNGP Kernel and the Matern Kernel

TL;DR

A surprising result is demonstrated that the predictions given from the NNGP kernel correspond closely to those given by the Matern kernel under specific circumstances, which suggests a deep similarity between overparameterized deep neural networks and the Matern kernel.

Abstract

Kernels representing limiting cases of neural network architectures have recently gained popularity. However, the application and performance of these new kernels compared to existing options, such as the Matern kernel, is not well studied. We take a practical approach to explore the neural network Gaussian process (NNGP) kernel and its application to data in Gaussian process regression. We first demonstrate the necessity of normalization to produce valid NNGP kernels and explore related numerical challenges. We further demonstrate that the predictions from this model are quite inflexible, and therefore do not vary much over the valid hyperparameter sets. We then demonstrate a surprising result that the predictions given from the NNGP kernel correspond closely to those given by the Matern kernel under specific circumstances, which suggests a deep similarity between overparameterized deep neural networks and the Matern kernel. Finally, we demonstrate the performance of the NNGP kernel as compared to the Matern kernel on three benchmark data cases, and we conclude that for its flexibility and practical performance, the Matern kernel is preferred to the novel NNGP in practical applications.

Figures (11)

• Figure 1: Comparison of kriging weights from a 1-dimensional Gaussian process regression compared on data normalized to the unit hypershpere and unnormalized.
• Figure 1: Marginal plots of the responses from the Friedman function against each input variable.
• Figure 2: a. Invalid NNGP kernel b. Valid NNGP Kernel
• Figure 2: a. Full training data b. Full testing data \newlabelfig:heaton0 . We subset 500 random samples from each of these for data in a single simulation iteration.
• Figure 3: Valid NNGP hyperparameter sets that produce positive definite kernel matrices for various depths for NNGP hyperparameters $\sigma_a$ and $\sigma_b$.