Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Bayesian Methods for a Tree-Structured Stick-Breaking Process Mixture of Gaussians by Application of the Bayes Codes for Context Tree Models

Yuta Nakahara

TL;DR

This paper proposes a learning algorithm with less computational cost for the TS-SBP mixture of Gaussians by using the VB method under an assumption of finite tree width and depth, and utilizes a subroutine in the Bayes coding algorithm for context tree models.

Abstract

The tree-structured stick-breaking process (TS-SBP) mixture model is a non-parametric Bayesian model that can represent tree-like hierarchical structures among the mixture components. For TS-SBP mixture models, only a Markov chain Monte Carlo (MCMC) method has been proposed and any variational Bayesian (VB) methods has not been proposed. In general, MCMC methods are computationally more expensive than VB methods. Therefore, we require a large computational cost to learn the TS-SBP mixture model. In this paper, we propose a learning algorithm with less computational cost for the TS-SBP mixture of Gaussians by using the VB method under an assumption of finite tree width and depth. When constructing such VB method, the main challenge is efficient calculation of a sum over all possible trees. To solve this challenge, we utilizes a subroutine in the Bayes coding algorithm for context tree models. We confirm the computational efficiency of our VB method through an experiments on a benchmark dataset.

Variational Bayesian Methods for a Tree-Structured Stick-Breaking Process Mixture of Gaussians by Application of the Bayes Codes for Context Tree Models

TL;DR

This paper proposes a learning algorithm with less computational cost for the TS-SBP mixture of Gaussians by using the VB method under an assumption of finite tree width and depth, and utilizes a subroutine in the Bayes coding algorithm for context tree models.

Abstract

The tree-structured stick-breaking process (TS-SBP) mixture model is a non-parametric Bayesian model that can represent tree-like hierarchical structures among the mixture components. For TS-SBP mixture models, only a Markov chain Monte Carlo (MCMC) method has been proposed and any variational Bayesian (VB) methods has not been proposed. In general, MCMC methods are computationally more expensive than VB methods. Therefore, we require a large computational cost to learn the TS-SBP mixture model. In this paper, we propose a learning algorithm with less computational cost for the TS-SBP mixture of Gaussians by using the VB method under an assumption of finite tree width and depth. When constructing such VB method, the main challenge is efficient calculation of a sum over all possible trees. To solve this challenge, we utilizes a subroutine in the Bayes coding algorithm for context tree models. We confirm the computational efficiency of our VB method through an experiments on a benchmark dataset.
Paper Structure (20 sections, 9 theorems, 46 equations, 5 figures)

This paper contains 20 sections, 9 theorems, 46 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Stochastic models
  3. Variational Bayesian methods
  4. Prior distributions
  5. Overview of variational Bayesian methods
  6. Update of $q(\bm \pi_s)$, $q(\bm \mu_s)$, $q(\bm \Lambda_s)$, $q(\bm L)$, $q(\bm g)$, and $q(\bm z_i)$
  7. Update of $q(T_i)$
  8. Initialization
  9. Experiments
  10. Toy example
  11. CIFAR-100 image dataset
  12. Conclusion
  13. Proof of Proposition \ref{['prop:q_other_T']}
  14. Update of $q(\bm \pi_s)$
  15. Update of $q(\bm \mu_s)$
  16. ...and 5 more sections

Key Result

Theorem 1

The probability distribution of $S_i$ over $\mathcal{S}_\mathrm{max}$ is represented as follows. where $s_\mathrm{pa}$ denotes the parent node of $s$ and we assume $\pi_{s_\mathrm{pa},s} = 1$ for $s=s_\lambda$. The above distribution is equivalent to a truncated version of the TS-SBP, where tree width and depth are $K$ and $D$, respectively.

Figures (5)

  • Figure 1: An overview of the data generation process.
  • Figure 2: Comparison of our defined TS-SBP with the original TS-SBP and the model used in 2023SMC from the perspective of graphical models.
  • Figure 3: The input data and the estimated tree structure of the means of the mixture components.
  • Figure 4: The TS-SBP mixture of Gaussians estimated from the data shown in Fig. \ref{['result_synthetic']}.
  • Figure 5: The images are placed at nodes with the maximum a posteriori probability. We only show the node with at least 50 images, and we show the 10 images with highest probabilities at each node.

Theorems & Definitions (13)

  • Definition 1
  • Definition 2: full_rooted_trees
  • Remark 1
  • Definition 3
  • Theorem 1
  • Proposition 1
  • Theorem 2
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • ...and 3 more