GPTreeO: An R package for continual regression with dividing local Gaussian processes

TL;DR

A sensitivity analysis is conducted to show how GPTreeO's configurable features impact the regression performance in a continual learning setting.

Abstract

We introduce GPTreeO, a flexible R package for scalable Gaussian process (GP) regression, particularly tailored to continual learning problems. GPTreeO builds upon the Dividing Local Gaussian Processes (DLGP) algorithm, in which a binary tree of local GP regressors is dynamically constructed using a continual stream of input data. In GPTreeO we extend the original DLGP algorithm by allowing continual optimisation of the GP hyperparameters, incorporating uncertainty calibration, and introducing new strategies for how the local partitions are created. Moreover, the modular code structure allows users to interface their favourite GP library to perform the local GP regression in GPTreeO. The flexibility of GPTreeO gives the user fine-grained control of the balance between computational speed, accuracy, stability and smoothness. We conduct a sensitivity analysis to show how GPTreeO's configurable features impact the regression performance in a continual learning setting.

Figures (4)

• Figure 1: Visual representation of how the splitting works. The tree structure is shown above. Reproduction of Figure 1 from LedererAl2020.
• Figure 2: An illustration of the gradual splitting approach to creating new nodes. When a node splits, two identical child nodes are created that each inherit the full dataset from the parent node (first and second panel). Each new data point is assigned to only one child node, and the chosen node simultaneously discards an old point that was shared with the other child node (third panel). Eventually, the two child nodes have no data points in common (fourth panel) and each can split into two new child nodes.
• Figure 3: Coverage of the $\mu \pm 1\sigma$ prediction in a GPTreeO test run, computed for batches of 2000 input points. The graphs show the results with (orange) and without (purple) using GPTreeO's uncertainty calibration option.
• Figure 4: Illustration of how the nature of the input stream affects the challenge of learning a given target function. Top: A GPTreeO test run with the four-dimensional Eggholder function as target function, where the stream of input points is generated through a differential evolution optimiser exploring a four-dimensional Rosenbrock function. Middle: A similar test setup as for the top panel, but now the stream of input points is generated by uniform sampling from the full domain. The points in the scatter plots are sorted such that the points with largest prediction errors are displayed on top. Bottom: The evolution of the prediction accuracy in the two test runs. The graphs show what fraction of predictions are within a given error threshold, computed for batches of 2000 input points. The green and purple graphs correspond to the test runs in the top and middle panels, respectively.