Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Laplace Variational Inference for Bayesian Envelope Models

Seunghyeon Kim, Kwangmin Lee, Yeonhee Park

Abstract

Envelope models provide a sufficient dimension reduction framework for multivariate regression analysis. Bayesian inference for these models has been developed primarily using Markov chain Monte Carlo (MCMC) methods. Specifically, Gibbs sampling and Metropolis-Hastings algorithms suffer from slow mixing and high computational cost. Although automatic differentiation variational inference (ADVI) has been explored for Bayesian envelope models, the resulting gradient-based optimization is often numerically unstable due to severe ill-conditioning of the posterior distribution. To address this issue, we propose a novel reparameterization of the posterior distribution that alleviates the ill-conditioning inherent in conventional variational approaches. Building on this reparameterization, we develop an efficient variational inference procedure. Since the resulting likelihood remains nonconjugate, we approximate the corresponding variational factor using a Laplace approximation within a coordinate-ascent variational inference (CAVI) framework. We establish theoretical results showing that, at each one-step coordinate update, the Laplace approximation error relative to the exact variational inference coordinate update converges to zero. Simulation studies and a real-data analysis demonstrate that the proposed method substantially improves computational efficiency while maintaining estimation accuracy and model-selection performance relative to existing approaches.

Laplace Variational Inference for Bayesian Envelope Models

Abstract

Envelope models provide a sufficient dimension reduction framework for multivariate regression analysis. Bayesian inference for these models has been developed primarily using Markov chain Monte Carlo (MCMC) methods. Specifically, Gibbs sampling and Metropolis-Hastings algorithms suffer from slow mixing and high computational cost. Although automatic differentiation variational inference (ADVI) has been explored for Bayesian envelope models, the resulting gradient-based optimization is often numerically unstable due to severe ill-conditioning of the posterior distribution. To address this issue, we propose a novel reparameterization of the posterior distribution that alleviates the ill-conditioning inherent in conventional variational approaches. Building on this reparameterization, we develop an efficient variational inference procedure. Since the resulting likelihood remains nonconjugate, we approximate the corresponding variational factor using a Laplace approximation within a coordinate-ascent variational inference (CAVI) framework. We establish theoretical results showing that, at each one-step coordinate update, the Laplace approximation error relative to the exact variational inference coordinate update converges to zero. Simulation studies and a real-data analysis demonstrate that the proposed method substantially improves computational efficiency while maintaining estimation accuracy and model-selection performance relative to existing approaches.
Paper Structure (52 sections, 17 theorems, 222 equations, 4 figures, 7 tables, 2 algorithms)

This paper contains 52 sections, 17 theorems, 222 equations, 4 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Review of Bayesian response envelope model
  3. Review of Laplace variational inference
  4. Coordinate-ascent variational inference.
  5. Conditionally conjugate blocks.
  6. Nonconjugate blocks and Laplace updates.
  7. CALVI for the Bayesian response envelope model
  8. Computational issues of parameterization (3)
  9. Reparameterization and CALVI algorithm
  10. Reparameterization.
  11. Key computational implication: the Laplace derivatives for $\boldsymbol{A}$.
  12. Laplace update for the nonconjugate block $\boldsymbol{A}$.
  13. Closed-form updates for the conjugate blocks.
  14. Algorithm summary and convergence monitoring.
  15. Theoretical justification of the CALVI Laplace update
  16. ...and 37 more sections

Key Result

Theorem 1

Suppose that Assumptions assum1--assum5 hold and adopt the notation above. Then

Figures (4)

  • Figure 1: Marginal posterior distributions for $\beta_{ij}, \ i=1,\dots,3 \ , j=1,\dots,3$ based on the first replicate of the simulated data. Red dotted lines denote the true values of the parameters. The marginal densities for the remaining coefficients can be seen in Figure \ref{['beta all1']} and Figure \ref{['beta all2']}.
  • Figure 2: Heatmaps of the regression coefficient matrices for each group, estimated under the response envelope model using the CALVI algorithm. Rows correspond to N-glycans (responses) and columns correspond to predictors. The color bar indicates the magnitude and direction of the coefficient values.
  • Figure S1: Marginal variational and posterior distributions for $\beta_{ij}, \ i=1,\dots,10 \ , j=1,\dots,7$ based on the first replicate of the simulated data. Red dotted lines denote the true values of the parameters.
  • Figure S2: Marginal variational and posterior distributions for $\beta_{ij}, \ i=11,\dots,20 \ , j=1,\dots,7$ based on the first replicate of the simulated data. Red dotted lines denote the true values of the parameters.

Theorems & Definitions (28)

  • Theorem 1
  • Theorem 2
  • Proposition 3: Exchange of differentiation and expectation (first order)
  • Lemma 4: Compact Jacobian for the inverse square-root factor
  • Lemma 5: Directional second derivative of the inverse square-root Jacobian
  • proof : Proof of Proposition \ref{['prop:DCT']}
  • Proposition 6: Kronecker--trace identity
  • Corollary 7
  • proof
  • Theorem 8
  • ...and 18 more