Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Infinite-dimensional next-generation reservoir computing

Lyudmila Grigoryeva, Hannah Lim Jing Ting, Juan-Pablo Ortega

TL;DR

This paper shows that NG-RC can be encoded as a kernel ridge regression that makes training efficient and feasible even when the space of chosen polynomial features is very large, which makes the methodology agnostic with respect to the lags into the past.

Abstract

Next-generation reservoir computing (NG-RC) has attracted much attention due to its excellent performance in spatio-temporal forecasting of complex systems and its ease of implementation. This paper shows that NG-RC can be encoded as a kernel ridge regression that makes training efficient and feasible even when the space of chosen polynomial features is very large. Additionally, an extension to an infinite number of covariates is possible, which makes the methodology agnostic with respect to the lags into the past that are considered as explanatory factors, as well as with respect to the number of polynomial covariates, an important hyperparameter in traditional NG-RC. We show that this approach has solid theoretical backing and good behavior based on kernel universality properties previously established in the literature. Various numerical illustrations show that these generalizations of NG-RC outperform the traditional approach in several forecasting applications.

Infinite-dimensional next-generation reservoir computing

TL;DR

This paper shows that NG-RC can be encoded as a kernel ridge regression that makes training efficient and feasible even when the space of chosen polynomial features is very large, which makes the methodology agnostic with respect to the lags into the past.

Abstract

Next-generation reservoir computing (NG-RC) has attracted much attention due to its excellent performance in spatio-temporal forecasting of complex systems and its ease of implementation. This paper shows that NG-RC can be encoded as a kernel ridge regression that makes training efficient and feasible even when the space of chosen polynomial features is very large. Additionally, an extension to an infinite number of covariates is possible, which makes the methodology agnostic with respect to the lags into the past that are considered as explanatory factors, as well as with respect to the number of polynomial covariates, an important hyperparameter in traditional NG-RC. We show that this approach has solid theoretical backing and good behavior based on kernel universality properties previously established in the literature. Various numerical illustrations show that these generalizations of NG-RC outperform the traditional approach in several forecasting applications.

Paper Structure

This paper contains 26 sections, 6 theorems, 114 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminary discussion
  3. Notation
  4. Data generating processes
  5. Kernel methods
  6. The NG-RC methodology
  7. Kernelization of NG-RC
  8. Infinite-dimensional NG-RC and Volterra kernels
  9. Numerics
  10. Data generating processes and experimental setup
  11. Time complexities
  12. Cross-validation
  13. Results
  14. Introduction
  15. Preliminary discussion
  16. ...and 11 more sections

Key Result

Lemma 2

Suppose $K_1$ and $K_2$ are kernels with feature spaces $H_1$ and $H_2$ and feature maps $\Phi_1 : \mathcal{Z} \rightarrow H_1$ and $\Phi_2 : \mathcal{Z} \rightarrow H_2$, respectively. Denote the corresponding RKHSs by $\mathbb{H}_1$ and $\mathbb{H}_2$. Suppose that there exists a bounded linear is

Figures (4)

  • Figure 1: The PSD for: (\ref{['fig: lorenz_volt_welch']}) the Lorenz system for the Volterra kernel, (\ref{['fig: lorenz_poly_welch']}) the polynomial kernel, and (\ref{['fig: lorenz_ngrc_welch']}) NG-RC. Dimension $1$ corresponds to $x$, dimension $2$ to $y$, and dimension $3$ to $z$. PSD for Mackey-Glass system for Volterra kernel is (\ref{['fig: mg_volt_welch']}), polynomial kernel is (\ref{['fig: mg_poly_welch']}), and NG-RC is (\ref{['fig: mg_ngrc_welch']}). Dimension 1 refers to the $z$ value. PSD for BEKK for Volterra kernel is (\ref{['fig: bekk_volt_welch']}), polynomial kernel is (\ref{['fig: bekk_poly_welch']}), and NG-RC is (\ref{['fig: bekk_ngrc_welch']}). Only the two dimensions with the most visually prominent PSD are displayed (dimensions 85 and 118).
  • Figure 2: The distributions for Lorenz system for the Volterra kernel is in (\ref{['fig: lorenz_volt_welch']}), the polynomial kernel is in (\ref{['fig: lorenz_poly_welch']}), and NG-RC is in (\ref{['fig: lorenz_ngrc_welch']}). The distribution for the Mackey-Glass system for Volterra kernel is (\ref{['fig: mg_volt_welch']}), the polynomial kernel is (\ref{['fig: mg_poly_welch']}), and NG-RC is (\ref{['fig: mg_ngrc_welch']}). The distributions for BEKK for Volterra kernel is (\ref{['fig: bekk_volt_welch']}), the polynomial kernel is (\ref{['fig: bekk_poly_welch']}), and NG-RC is (\ref{['fig: bekk_ngrc_welch']}). Only one dimension is displayed (dimension 118).
  • Figure 3: The PSD for: (\ref{['fig: lorenz_volt_welch']}) the Lorenz system for the Volterra kernel, (\ref{['fig: lorenz_poly_welch']}) the polynomial kernel, and (\ref{['fig: lorenz_ngrc_welch']}) NG-RC. Dimension $1$ corresponds to $x$, dimension $2$ to $y$, and dimension $3$ to $z$. PSD for Mackey-Glass system for Volterra kernel is (\ref{['fig: mg_volt_welch']}), polynomial kernel is (\ref{['fig: mg_poly_welch']}), and NG-RC is (\ref{['fig: mg_ngrc_welch']}). Dimension 1 refers to the $z$ value. PSD for BEKK for Volterra kernel is (\ref{['fig: bekk_volt_welch']}), polynomial kernel is (\ref{['fig: bekk_poly_welch']}), and NG-RC is (\ref{['fig: bekk_ngrc_welch']}). Only the two dimensions with the most visually prominent PSD are displayed (dimensions 85 and 118).
  • Figure 4: The distributions for Lorenz system for the Volterra kernel is in (\ref{['fig: lorenz_volt_welch']}), the polynomial kernel is in (\ref{['fig: lorenz_poly_welch']}), and NG-RC is in (\ref{['fig: lorenz_ngrc_welch']}). The distribution for the Mackey-Glass system for Volterra kernel is (\ref{['fig: mg_volt_welch']}), the polynomial kernel is (\ref{['fig: mg_poly_welch']}), and NG-RC is (\ref{['fig: mg_ngrc_welch']}). The distributions for BEKK for Volterra kernel is (\ref{['fig: bekk_volt_welch']}), the polynomial kernel is (\ref{['fig: bekk_poly_welch']}), and NG-RC is (\ref{['fig: bekk_ngrc_welch']}). Only one dimension is displayed (dimension 118).

Theorems & Definitions (14)

  • Example 1: Polynomial kernels
  • Lemma 2
  • proof
  • Proposition 3
  • proof
  • Example 4
  • Theorem 5
  • Example 6: Polynomial kernels
  • Lemma 7
  • proof
  • ...and 4 more