Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Variational Inference for Mixed Data Mixture Models

Junyang Wang, James Bennett, Victor Lhoste, Sarah Filippi

TL;DR

A coordinate ascent variational inference algorithm (CAVI) for mixture models on mixed (continuous and categorical) data, which circumvents the high computational cost of MCMC while retaining uncertainty quantification.

Abstract

Heterogeneous, mixed type datasets including both continuous and categorical variables are ubiquitous, and enriches data analysis by allowing for more complex relationships and interactions to be modelled. Mixture models offer a flexible framework for capturing the underlying heterogeneity and relationships in mixed type datasets. Most current approaches for modelling mixed data either forgo uncertainty quantification and only conduct point estimation, and some use MCMC which incurs a very high computational cost that is not scalable to large datasets. This paper develops a coordinate ascent variational inference algorithm (CAVI) for mixture models on mixed (continuous and categorical) data, which circumvents the high computational cost of MCMC while retaining uncertainty quantification. We demonstrate our approach through simulation studies as well as an applied case study of the NHANES risk factor dataset. We provide theoretical justification for our method by establishing that the CAVI variational posterior mean converges locally to the true parameter value at a gap of $O(1/n)$ from the maximum likelihood estimator. Building on this result, we show that the CAVI variational posterior contracts around the true parameter at $O(n^{-1/2})$ rate.

Bayesian Variational Inference for Mixed Data Mixture Models

TL;DR

A coordinate ascent variational inference algorithm (CAVI) for mixture models on mixed (continuous and categorical) data, which circumvents the high computational cost of MCMC while retaining uncertainty quantification.

Abstract

Heterogeneous, mixed type datasets including both continuous and categorical variables are ubiquitous, and enriches data analysis by allowing for more complex relationships and interactions to be modelled. Mixture models offer a flexible framework for capturing the underlying heterogeneity and relationships in mixed type datasets. Most current approaches for modelling mixed data either forgo uncertainty quantification and only conduct point estimation, and some use MCMC which incurs a very high computational cost that is not scalable to large datasets. This paper develops a coordinate ascent variational inference algorithm (CAVI) for mixture models on mixed (continuous and categorical) data, which circumvents the high computational cost of MCMC while retaining uncertainty quantification. We demonstrate our approach through simulation studies as well as an applied case study of the NHANES risk factor dataset. We provide theoretical justification for our method by establishing that the CAVI variational posterior mean converges locally to the true parameter value at a gap of from the maximum likelihood estimator. Building on this result, we show that the CAVI variational posterior contracts around the true parameter at rate.

Paper Structure

This paper contains 29 sections, 9 theorems, 155 equations, 9 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Mixed data mixture model
  3. Variational Inference
  4. Background on Variational Inference and Coordinate Ascent
  5. CAVI for mixed data mixture model
  6. Posterior Predictive Distribution
  7. Theory
  8. Experimental Results
  9. An illustrative one-dimensional example
  10. Simulation study
  11. Simulation set-ups and error metrics
  12. Summary of simulation study results
  13. Application: Identifying groups of individuals with comparable health phenotypes based on survey responses
  14. Discussion
  15. Coordinate Ascent Variational Inference update equations
  16. ...and 14 more sections

Key Result

Theorem 1

Almost surely as the sample size $n \rightarrow \infty$, the iterative procedure defined in eq: iterativeprocedure converges locally to the true data generating parameter $\Theta^{*}$ whenever $0 < \epsilon <2$. Locally here means whenever the starting values are sufficiently near $\Theta^{*}$.

Figures (9)

  • Figure 1: Plot of marginal posterior distribution of $\mu$ and $\Sigma$ for both variational inference and Gibbs
  • Figure 2: Plot of marginal posterior distribution of $\psi_{1,g}, \psi_{2,g}$ for both variational inference and Gibbs
  • Figure 3: Plot of marginal posterior distribution of $\psi_{3,g}, \pi$ for both variational inference and Gibbs
  • Figure 4: Comparison of the variational posterior predictive distribution of $x$ given $c$, and $c$ to the empirical data and the likelihood function evaluated at the data generating parameters.
  • Figure 5: Radar plots of the marginal posterior predictive distribution of each continuous variable, conditioned on cluster component. The posterior mean of each $\pi_k$ is also added atop each radar plot. The concentric circles and quantiles in each radar plot represents the quantiles of the empirical distribution of the whole data. For the sake of visual consistency, the distribution of height, eGFR and hdl are reversed so that higher values in the radar plot for all continuous variables correspond to higher risk of chronic diseases.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Theorem 1
  • Corollary 2
  • Corollary 3
  • Corollary 4
  • Corollary 5
  • Lemma D.1: Lemma 1 of BoWang2006
  • Lemma D.2
  • Lemma D.3
  • Lemma D.4