Analytical Approximation of the ELBO Gradient in the Context of the Clutter Problem

TL;DR

This work addresses the challenge of computing ELBO gradients analytically in the clutter problem, where observations arise from a Gaussian component embedded in unknown clutter. By leveraging the reparameterization trick to move the gradient inside expectations and employing local Taylor-based Gaussian approximations of the likelihood factors, the authors derive tractable, closed-form gradient expressions. These gradients are integrated into an EM framework, yielding linear-time update rules for the variational mean and variance and enabling fast, deterministic inference suitable for real-time or edge applications. The method is evaluated against Laplace, EP, and MFVI, demonstrating competitive accuracy and convergence behavior, with clear pathways for extension to multidimensional data and non-Gaussian variants through Gaussian mixtures. Overall, the approach provides a principled, scalable alternative for ELBO optimization in cluttered Bayesian models with potential practical impact on sensing and safety-critical systems.

Abstract

We propose an analytical solution for approximating the gradient of the Evidence Lower Bound (ELBO) in variational inference problems where the statistical model is a Bayesian network consisting of observations drawn from a mixture of a Gaussian distribution embedded in unrelated clutter, known as the clutter problem. The method employs the reparameterization trick to move the gradient operator inside the expectation and relies on the assumption that, because the likelihood factorizes over the observed data, the variational distribution is generally more compactly supported than the Gaussian distribution in the likelihood factors. This allows efficient local approximation of the individual likelihood factors, which leads to an analytical solution for the integral defining the gradient expectation. We integrate the proposed gradient approximation as the expectation step in an EM (Expectation Maximization) algorithm for maximizing ELBO and test against classical deterministic approaches in Bayesian inference, such as the Laplace approximation, Expectation Propagation and Mean-Field Variational Inference. The proposed method demonstrates good accuracy and rate of convergence together with linear computational complexity.

Figures (5)

• Figure 1: Local approximation of the reparameterized likelihood factors
• Figure 2: Analytical approximation of the ELBO gradient over the mean $\mu_q$ (left panel) and variance $v_q$ (right panel) of the variational distribution
• Figure 3: Testing the ELBO gradient analytical approximation (ELBO GAA) against classical deterministic approaches for a sample size of 20 data points
• Figure 4: Testing the proposed approximation (ELBO GAA) against classical deterministic approaches for sample sizes of 5 (left), 10 (middle) and 100 (right) data points
• Figure 5: Execution times for the proposed method (ELBO GAA) in comparison to classical deterministic approaches and stochastic gradient descent (SGD) for 20 data points with serial execution (left panel) and vectorization (right panel)