Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBM

Nikhil Kapasi, William Whitehead, Luke Theogarajan

TL;DR

The paper introduces the Gaussian-Multinoulli RBM (GM-RBM), an energy-based extension of the Gaussian–Bernoulli RBM that replaces binary hidden units with $q$-state Potts variables to achieve a discrete, interpretable latent space while modeling Gaussian-visible data. It derives the GM-RBM energy, conditional distributions, and compositional latent structure, demonstrating how the mean visible vector $\mu(h)$ is a sum of state-specific templates $W_j^{(h_j)}$. Empirical results on hetero- and auto-associative tasks show that higher Potts state counts $q$ improve recall accuracy and sample quality under fixed parameter budgets, with $q=4$ often offering a favorable trade-off between capacity and efficiency. The work discusses limitations (e.g., reliance on Gibbs sampling) and outlines future directions, including scaling to deeper architectures, improved discrete sampling, and hardware-oriented implementations, highlighting Potts units as a route to more expressive and scalable discrete latent representations in energy-based models.

Abstract

Many real-world tasks, from associative memory to symbolic reasoning, demand discrete, structured representations that standard continuous latent models struggle to express naturally. We introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian-Bernoulli RBM (GB-RBM) by replacing binary hidden units with $q$-state Potts variables. This modification enables a combinatorially richer latent space and supports learning over multivalued, interpretable latent concepts. We formally derive GM-RBM's energy function, learning dynamics, and conditional distributions, showing that it preserves tractable inference and training through contrastive divergence. Empirically, we demonstrate that GM-RBMs model complex multimodal distributions more effectively than binary RBMs, outperforming them on tasks involving analogical recall and structured memory. Our results highlight GM-RBMs as a scalable framework for discrete latent inference with enhanced expressiveness and interoperability.

The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBM

TL;DR

The paper introduces the Gaussian-Multinoulli RBM (GM-RBM), an energy-based extension of the Gaussian–Bernoulli RBM that replaces binary hidden units with -state Potts variables to achieve a discrete, interpretable latent space while modeling Gaussian-visible data. It derives the GM-RBM energy, conditional distributions, and compositional latent structure, demonstrating how the mean visible vector is a sum of state-specific templates . Empirical results on hetero- and auto-associative tasks show that higher Potts state counts improve recall accuracy and sample quality under fixed parameter budgets, with often offering a favorable trade-off between capacity and efficiency. The work discusses limitations (e.g., reliance on Gibbs sampling) and outlines future directions, including scaling to deeper architectures, improved discrete sampling, and hardware-oriented implementations, highlighting Potts units as a route to more expressive and scalable discrete latent representations in energy-based models.

Abstract

Many real-world tasks, from associative memory to symbolic reasoning, demand discrete, structured representations that standard continuous latent models struggle to express naturally. We introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian-Bernoulli RBM (GB-RBM) by replacing binary hidden units with -state Potts variables. This modification enables a combinatorially richer latent space and supports learning over multivalued, interpretable latent concepts. We formally derive GM-RBM's energy function, learning dynamics, and conditional distributions, showing that it preserves tractable inference and training through contrastive divergence. Empirically, we demonstrate that GM-RBMs model complex multimodal distributions more effectively than binary RBMs, outperforming them on tasks involving analogical recall and structured memory. Our results highlight GM-RBMs as a scalable framework for discrete latent inference with enhanced expressiveness and interoperability.
Paper Structure (30 sections, 7 equations, 3 figures, 2 tables)

This paper contains 30 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Model Overview
  3. Background and Motivation
  4. Motivating Discrete Latent Structure: Potts Units
  5. The Gaussian-Multinoulli RBM (GM-RBM)
  6. Theoretical Foundations of the Gaussian-Multinoulli RBM (GM-RBM)
  7. Energy Function and Architecture
  8. Compositionality and Factored Representation
  9. Hetero-associative Memory
  10. Experimental Setup
  11. Dataset
  12. Preprocessing
  13. Computational Setup
  14. Training
  15. Results
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Retrieval accuracy versus the number of associative pairs for different numbers of Potts states (q) in a parameter-matched GB-RBM and GM-RBM setup
  • Figure 2: Retrieval accuracy on a semantic hetero‐associative memory task as a function of hidden‐layer size for different model variants and dataset sizes. (a) GB-RBM (Gibbs--Langevin Update), (b) GM-RBM with $q=2$ (Gibbs Update), and (c) GM-RBM with $q=4$ (Gibbs Update). Each curve corresponds to a different number of training word‐pair examples ($N$).
  • Figure 3: Sampled results from GM-RBM of MNIST and CelebA datatsets