Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel

Ziyang Jiang, Tongshu Zheng, Yiling Liu, David Carlson

TL;DR

This work tackles the challenge of injecting prior knowledge into neural networks by blending an NN-implied kernel with a chosen low-dimensional kernel within a Gaussian-process framework. The proposed Implicit Composite Kernel (ICK) maps the low-dimensional kernel into a latent space via Nyström or Random Fourier Features, enabling the neural network to couple high-dimensional information with principled prior structure. The authors show, under reasonable assumptions, that ICK approximates sampling from a GP with a multiplicative composite kernel, and they offer two uncertainty estimation routes: deep ensembles and direct GP posterior variance using the Nyström-based mapping. Through synthetic and real-data experiments, including remote sensing tasks, ICK demonstrates superior predictive performance and flexible incorporation of prior patterns (e.g., seasonality) with scalable learning. Limitations include performance degradation when integrating many sources and potential instability in kernel-based posterior estimates, but overall ICK provides a practical, scalable pathway to hybrid, knowledge-informed neural modeling.

Abstract

It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known properties, such as spatial smoothness or seasonality, are straightforward to model by choosing an appropriate kernel in a Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolutional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties (e.g., seasonality). We implement this idea by combining a deep network and an efficient mapping based on the Nystrom approximation, which we call Implicit Composite Kernel (ICK). We then adopt a sample-then-optimize approach to approximate the full GP posterior distribution. We demonstrate that ICK has superior performance and flexibility on both synthetic and real-world data sets. We believe that ICK framework can be used to include prior information into neural networks in many applications.

Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel

TL;DR

This work tackles the challenge of injecting prior knowledge into neural networks by blending an NN-implied kernel with a chosen low-dimensional kernel within a Gaussian-process framework. The proposed Implicit Composite Kernel (ICK) maps the low-dimensional kernel into a latent space via Nyström or Random Fourier Features, enabling the neural network to couple high-dimensional information with principled prior structure. The authors show, under reasonable assumptions, that ICK approximates sampling from a GP with a multiplicative composite kernel, and they offer two uncertainty estimation routes: deep ensembles and direct GP posterior variance using the Nyström-based mapping. Through synthetic and real-data experiments, including remote sensing tasks, ICK demonstrates superior predictive performance and flexible incorporation of prior patterns (e.g., seasonality) with scalable learning. Limitations include performance degradation when integrating many sources and potential instability in kernel-based posterior estimates, but overall ICK provides a practical, scalable pathway to hybrid, knowledge-informed neural modeling.

Abstract

It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known properties, such as spatial smoothness or seasonality, are straightforward to model by choosing an appropriate kernel in a Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolutional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties (e.g., seasonality). We implement this idea by combining a deep network and an efficient mapping based on the Nystrom approximation, which we call Implicit Composite Kernel (ICK). We then adopt a sample-then-optimize approach to approximate the full GP posterior distribution. We demonstrate that ICK has superior performance and flexibility on both synthetic and real-world data sets. We believe that ICK framework can be used to include prior information into neural networks in many applications.
Paper Structure (44 sections, 2 theorems, 28 equations, 16 figures, 7 tables, 3 algorithms)

This paper contains 44 sections, 2 theorems, 28 equations, 16 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Equivalence between NNs and GPs
  4. NNs with prior knowledge
  5. GP with composite kernels
  6. GP for large datasets
  7. Background
  8. Problem Setup
  9. Composite GPs
  10. Correspondence between GPs and NNs
  11. Implicit Composite Kernel
  12. Resemblance between Composite GPR and ICK
  13. Kernel-to-latent-space Transformation
  14. Nyström Approximation
  15. Random Fourier Features
  16. ...and 29 more sections

Key Result

Theorem 1

Let $f_{\text{NN}}: \mathbb{R}^{D_1} \rightarrow \mathbb{R}^p$ be a NN function with random weights and $g: \mathbb{R}^{D_2} \rightarrow \mathbb{R}^p$ be a mapping function, and define an inner product between the representations $\hat{y} = f_{\text{ICK}} \left( \boldsymbol{x}^{(1)}, \boldsymbol{x}^ if $f_{\text{NN}}$ is a neural network with zero-mean i.i.d. parameters and continuous activation f

Figures (16)

  • Figure 1: Given data containing 2 sources of information $\boldsymbol{x}^{(1)}$ and $\boldsymbol{x}^{(2)}$, we can process the data using either (Left) a composite Gaussian process regression (GPR) model or (Right) our ICK framework where $\boldsymbol{x}^{(1)}$ is processed with a neural network $f_{\text{NN}}(\cdot)$ and $\boldsymbol{x}^{(2)}$ is processed with $g(\cdot)$ where $g(\cdot)$ consists of a kernel function $K_2$ and some transformation which maps the kernel matrix $\boldsymbol{K}_2$ into the latent space.
  • Figure 2: Given data containing $M$ sources of information $\boldsymbol{x} = \left\{ x^{(1)}, x^{(2)}, ..., x^{(M)} \right\}$, we can process the data using our ICK framework where high-dimensional information (e.g. $x^{(1)}$ in the figure) is processed using a neural network and low-dimensional information (e.g. $x^{(2)}$ in the figure) is processed using a kernel function and some transformation which maps the kernel matrix into the latent space.
  • Figure 3: Eigenvalues of the kernel matrix computed from the first 4 batches of training data where $N$ is the total number of data points.
  • Figure 4: Predictive distribution from ICKy ensemble and its GP posterior counterpart on a 1D regression task.
  • Figure 5: Plots of predicted mean and uncertainties for a sinusoidal function with exponential damping with noise $f(x) = e^{-x} \sin(2\pi x) + \epsilon$ using (a) an exact GP with RBF kernel and (b) ICKy with RBF kernel. Here ICKy predicts the uncertainty by directly calculating the covariance matrix $\boldsymbol{\Sigma}$ in Equation \ref{['eq:17']}.
  • ...and 11 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Lemma 2