Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vectorized Bayesian Inference for Latent Dirichlet-Tree Allocation

Zheng Wang, Nizar Bouguila

TL;DR

The framework of Latent Dirichlet-Tree Allocation (LDTA), a generalization of LDA that replaces the Dirichlet prior with an arbitrary Dirichlet-Tree (DT) distribution, is introduced, which preserves LDA's generative structure but enables expressive, tree-structured priors over topic proportions.

Abstract

Latent Dirichlet Allocation (LDA) is a foundational model for discovering latent thematic structure in discrete data, but its Dirichlet prior cannot represent the rich correlations and hierarchical relationships often present among topics. We introduce the framework of Latent Dirichlet-Tree Allocation (LDTA), a generalization of LDA that replaces the Dirichlet prior with an arbitrary Dirichlet-Tree (DT) distribution. LDTA preserves LDA's generative structure but enables expressive, tree-structured priors over topic proportions. To perform inference, we develop universal mean-field variational inference and Expectation Propagation, providing tractable updates for all DT. We reveal the vectorized nature of the two inference methods through theoretical development, and perform fully vectorized, GPU-accelerated implementations. The resulting framework substantially expands the modeling capacity of LDA while maintaining scalability and computational efficiency.

Vectorized Bayesian Inference for Latent Dirichlet-Tree Allocation

TL;DR

The framework of Latent Dirichlet-Tree Allocation (LDTA), a generalization of LDA that replaces the Dirichlet prior with an arbitrary Dirichlet-Tree (DT) distribution, is introduced, which preserves LDA's generative structure but enables expressive, tree-structured priors over topic proportions.

Abstract

Latent Dirichlet Allocation (LDA) is a foundational model for discovering latent thematic structure in discrete data, but its Dirichlet prior cannot represent the rich correlations and hierarchical relationships often present among topics. We introduce the framework of Latent Dirichlet-Tree Allocation (LDTA), a generalization of LDA that replaces the Dirichlet prior with an arbitrary Dirichlet-Tree (DT) distribution. LDTA preserves LDA's generative structure but enables expressive, tree-structured priors over topic proportions. To perform inference, we develop universal mean-field variational inference and Expectation Propagation, providing tractable updates for all DT. We reveal the vectorized nature of the two inference methods through theoretical development, and perform fully vectorized, GPU-accelerated implementations. The resulting framework substantially expands the modeling capacity of LDA while maintaining scalability and computational efficiency.
Paper Structure (45 sections, 11 theorems, 159 equations, 22 figures, 2 tables, 5 algorithms)

This paper contains 45 sections, 11 theorems, 159 equations, 22 figures, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Dirichlet-Tree
  3. Tree
  4. Hierarchical Multinomial and Dirichlet-Tree
  5. Exponential Form and Conjugacy
  6. Bayesian operator
  7. Derived Distributions of Dirichlet-Tree
  8. Latent Dirichlet-Tree Allocation
  9. The Generative Process
  10. The Joint and the Marginalized Distributions
  11. Mean-Field Variational Inference
  12. The Evidence Lower Bound
  13. E-Step
  14. M-Step
  15. Expectation Propagation
  16. ...and 30 more sections

Key Result

Theorem 1

In a tree structure, for any internal node $s \in \boldsymbol{\Lambda}$,

Figures (22)

  • Figure 1: Notation for a general tree structure
  • Figure 2: (a) Conventional Multinomial; (b) Hierarchical Multinomial; (c) Dirichlet-Tree
  • Figure 3: The document--topic--word counting tensor.
  • Figure 4: Graphical representation (plate notation) of LDTA model. The shaded circle represents the observed variable; the blank circles represent the latent variables and diamonds indicate hyper-parameters. The arrows imply dependent generative relationship and the rectangles with number at its right-lower corner represent repetitions.
  • Figure 5: Tensor expression of relationship of the variables
  • ...and 17 more figures

Theorems & Definitions (15)

  • Theorem 1
  • Definition 2: General form of Dirichlet-Tree
  • Theorem 3: Exponential form of Dirichlet-Tree
  • Corollary 4: Expectation of sufficient statistics
  • Corollary 5: Log-space expectation
  • Definition 6: Bayesian operator
  • Theorem 7: The central theorem
  • Corollary 8: Expectation operator
  • Definition 9: Derived distributions of Dirichlet-Tree
  • Corollary 10: Expectation matrix of derived distributions
  • ...and 5 more