Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building Expressive and Tractable Probabilistic Generative Models: A Review

Sahil Sidheekh, Sriraam Natarajan

TL;DR

This survey addresses the challenge of building expressive yet tractable probabilistic generative models by focusing on Probabilistic Circuits (PCs) as a unifying framework. It surveys foundational properties (smoothness, decomposability, determinism), learning algorithms (gradient-based optimization and EM), and structure-learning approaches (heuristic and Bayesian) for PCs, along with advances in deep and hybrid PCs that integrate neural networks, VAEs, and normalizing flows. The authors present a taxonomy of PC extensions and a discussion of how these models can achieve exact or efficient inference for a broad class of queries, and they outline open problems and promising directions for bridging DGMs and PCs. The practical impact lies in enabling scalable, interpretable probabilistic reasoning with exact computations for complex tasks, while guiding future work toward richer latent representations, adversarial training, multi-modal data, and cross-domain applications with strong theoretical foundations.

Abstract

We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.

Building Expressive and Tractable Probabilistic Generative Models: A Review

TL;DR

This survey addresses the challenge of building expressive yet tractable probabilistic generative models by focusing on Probabilistic Circuits (PCs) as a unifying framework. It surveys foundational properties (smoothness, decomposability, determinism), learning algorithms (gradient-based optimization and EM), and structure-learning approaches (heuristic and Bayesian) for PCs, along with advances in deep and hybrid PCs that integrate neural networks, VAEs, and normalizing flows. The authors present a taxonomy of PC extensions and a discussion of how these models can achieve exact or efficient inference for a broad class of queries, and they outline open problems and promising directions for bridging DGMs and PCs. The practical impact lies in enabling scalable, interpretable probabilistic reasoning with exact computations for complex tasks, while guiding future work toward richer latent representations, adversarial training, multi-modal data, and cross-domain applications with strong theoretical foundations.

Abstract

We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.
Paper Structure (32 sections, 2 figures)

This paper contains 32 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Notation.
  3. Tractable Probabilistic Modeling
  4. Inference Queries
  5. Enforcing Tractability by Imposing Structure
  6. Probabilistic Circuits
  7. Structural Properties for Probabilistic Circuits
  8. Parameter Learning
  9. Gradient Based Optimization.
  10. Expectation Maximization.
  11. Structure Learning
  12. Heuristics Based.
  13. Bayesian Approaches.
  14. Deep PCs via Random Structures
  15. Probabilistic Circuits vs Neural Networks.
  16. ...and 17 more sections

Figures (2)

  • Figure 1: A broad taxonomy of the literature on improving probabilistic circuits via better learning algorithms and design extensions.
  • Figure 2: A taxonomy of the literature on building hybrid models to bridge the gap between deep generative models and probabilistic circuits

Theorems & Definitions (4)

  • Definition 1
  • Definition 2: Smoothness
  • Definition 3: Decomposability
  • Definition 4: Determinism