A Kernel Approach for Semi-implicit Variational Inference

TL;DR

This work tackles the intractability of densities in semi-implicit variational inference by introducing Kernel Semi-Implicit Variational Inference (KSIVI), which leverages reproducing kernel Hilbert spaces to obtain an explicit optimal denoiser and reduces training to a kernel Stein discrepancy objective. By exploiting the semi-implicit hierarchical structure, KSIVI achieves a tractable, gradient-based optimization with variance bounds and non-asymptotic generalization guarantees. The approach is extended to multi-layer Hierarchical KSIVI (HKSIVI) and assessed across synthetic tasks, Bayesian logistic regression, conditioned diffusion, and Bayesian neural networks, demonstrating competitive performance and improved stability relative to SIVI-SM. The paper also investigates kernel choices for heavy-tailed targets and provides a theoretical framework for optimization and generalization in this kernelized VI setting, highlighting practical implications for scalable Bayesian inference.

Abstract

Semi-implicit variational inference (SIVI) enhances the expressiveness of variational families through hierarchical semi-implicit distributions, but the intractability of their densities makes standard ELBO-based optimization biased. Recent score-matching approaches to SIVI (SIVI-SM) address this issue via a minimax formulation, at the expense of an additional lower-level optimization problem. In this paper, we propose kernel semi-implicit variational inference (KSIVI), a principled and tractable alternative that eliminates the lower-level optimization by leveraging kernel methods. We show that when optimizing over a reproducing kernel Hilbert space, the lower-level problem admits an explicit solution, reducing the objective to the kernel Stein discrepancy (KSD). Exploiting the hierarchical structure of semi-implicit distributions, the resulting KSD objective can be efficiently optimized using stochastic gradient methods. We establish optimization guarantees via variance bounds on Monte Carlo gradient estimators and derive statistical generalization bounds of order $\tilde{\mathcal{O}}(1/\sqrt{n})$. We further introduce a multi-layer hierarchical extension that improves expressiveness while preserving tractability. Empirical results on synthetic and real-world Bayesian inference tasks demonstrate the effectiveness of KSIVI.

Figures (15)

• Figure 1: Gradient norm comparison across sample sizes for vanilla and U-statistic estimators. The shaded areas represent the standard deviations, estimated from 50 independent runs.
• Figure 2: Performances of KSIVI on toy examples. The histplots in blue represent the estimated densities using 100,000 samples generated from KSIVI's variational approximation. The black lines depict the contour of the target distributions.
• Figure 3: Convergence of KL divergence during training obtained by different methods on toy examples. The KL divergences are estimated using the Python ITE module ITE2014 with 100,000 samples. The results are averaged over 5 independent computations with the standard deviation as the shaded region.
• Figure 4: Marginal and pairwise variational approximations of $\beta_2,\beta_3,\beta_4$ on the Bayesian logistic regression task. The contours of the pairwise posterior approximation produced by SIVI-SM (in orange), SIVI (in green), and KSIVI (in blue) are graphed in comparison to the ground truth (in black). The sample size is 1000.
• Figure 5: Comparison between the estimated pairwise correlation coefficients and the ground truth on the Bayesian logistic regression task. Each scatter represents the estimated correlation coefficient ($y$-axis) and the ground truth correlation coefficient ($x$-axis) of some pair $(\beta_i,\beta_j)$. The lines in the same color as the scatters represent the regression lines. The sample size is 1000.

Theorems & Definitions (20)

• Theorem 3.1
• Proposition 3.2
• Proposition 3.3
• Proposition 4.1
• Theorem 4.2
• Theorem 4.3
• Theorem 4.4
• Theorem 4.5
• Corollary 4.6
• Theorem A.1