Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes

TL;DR

This work adopts natural gradient methods from information geometry for variational parameter optimization of Student-t Processes, utilizing tools such as the Fisher information matrix which is linked to the Beta function in the model.

Abstract

Recently, a sparse version of Student-t Processes, termed sparse variational Student-t Processes, has been proposed to enhance computational efficiency and flexibility for real-world datasets using stochastic gradient descent. However, traditional gradient descent methods like Adam may not fully exploit the parameter space geometry, potentially leading to slower convergence and suboptimal performance. To mitigate these issues, we adopt natural gradient methods from information geometry for variational parameter optimization of Student-t Processes. This approach leverages the curvature and structure of the parameter space, utilizing tools such as the Fisher information matrix which is linked to the Beta function in our model. This method provides robust mathematical support for the natural gradient algorithm when using Student's t-distribution as the variational distribution. Additionally, we present a mini-batch algorithm for efficiently computing natural gradients. Experimental results across four benchmark datasets demonstrate that our method consistently accelerates convergence speed.

Figures (2)

• Figure 1: Negative ELBO Curves for the Four Datasets
• Figure 2: Test MSE Curves for the Four Datasets

Theorems & Definitions (4)

• Definition 1
• Definition 2
• Lemma 1
• Lemma 2