nuGPR: GPU-Accelerated Gaussian Process Regression with Iterative Algorithms and Low-Rank Approximations

TL;DR

nuGPR addresses the prohibitive training cost of Gaussian Process Regression by integrating a suite of GPU-accelerated numerical techniques. The approach centers on preconditioned conjugate gradient solves, a Padé-based log-determinant via trace estimation, and a cluster-driven low-rank approximation of off-diagonal covariance blocks, all implemented natively in CUDA. Numerical gradients replace backpropagation for hyperparameter optimization, further reducing overhead. Empirical results show up to 2x faster training and 12x lower memory usage compared to a strong GPU baseline (GPyTorch) on synthetic and real-world datasets, with RMSE staying competitive, highlighting practical scalability of GPR on a single GPU.

Abstract

Gaussian Process Regression (GPR) is an important type of supervised machine learning model with inherent uncertainty measure in its predictions. We propose a new framework, nuGPR, to address the well-known challenge of high computation cost associated with GPR training. Our framework includes several ideas from numerical linear algebra to reduce the amount of computation in key steps of GPR, and we combine them to establish an end-to-end training algorithm. Specifically, we leverage the preconditioned conjugate gradient method to accelerate the convergence of the linear solves required in GPR. We exploit clustering in the input data to identify block-diagonal structure of the covariance matrix and subsequently construct low-rank approximations of the off-diagonal blocks. These enhancements significantly reduce the time and space complexity of our computations. In addition, unlike other frameworks that rely on exact differentiation, we employ numerical gradients to optimize the hyperparameters of our GPR model, further reducing the training cost by eliminating the need for backpropagation. Lastly, we leverage the CUDA Toolkit to efficiently parallelize the training procedure on NVIDIA GPUs. As a result, nuGPR reduces total training time by up to 2x and peak memory consumption by up to 12x on various synthetic and real-world datasets when compared to the best existing GPU-based GPR implementation.

Figures (6)

• Figure 1: Results of synthetic 1-D dataset defined in \ref{['eq1d']}
• Figure 2: Results of synthetic 3-D dataset defined in \ref{['eq3d']}
• Figure 3: Results of real-world datasets (Kin40k, Gas Sensors, MNIST)
• Figure 4: GP predictions for synthetic 1-D dataset. The green bold lines denote the mean, and the pink shaded areas denote the confidence region. The nuGPR and GPyTorch frameworks result in nearly identical predictions.
• Figure 5: Covariance matrices after 50 epochs for Kin40k dataset. The matrix from nuGPR contains unchanged diagonal blocks and approximated off-diagonal blocks. The overall pattern in the original matrix is preserved well.