Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating Inductive Biases to Energy-based Generative Models

Yukun Li, Li-Ping Liu

TL;DR

This work introduces a hybrid energy-based model that injects inductive bias by adding a fixed statistic term to the energy, yielding a tractable exponential-family formulation in the statistic weights. Trained with score matching across multiple noise levels, the model approximately maximizes data likelihood while aligning model and data statistics via $\mathbb{E}_{p_{\theta,\boldsymbol{\eta}}}[\mathbf{T}(\mathbf{x})] \approx \mathbb{E}_{data}[\mathbf{T}(\mathbf{x})]$. Empirical results across molecular graphs, handwritten digits, and point clouds demonstrate improved data fitting and generation when informative statistics—such as valency constraints, image border priors, and surface smoothness—are incorporated. The approach enables explicit incorporation of domain knowledge during training, offering a principled way to constrain and steer generative models toward desirable data properties with broad applicability.

Abstract

With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.

Incorporating Inductive Biases to Energy-based Generative Models

TL;DR

This work introduces a hybrid energy-based model that injects inductive bias by adding a fixed statistic term to the energy, yielding a tractable exponential-family formulation in the statistic weights. Trained with score matching across multiple noise levels, the model approximately maximizes data likelihood while aligning model and data statistics via . Empirical results across molecular graphs, handwritten digits, and point clouds demonstrate improved data fitting and generation when informative statistics—such as valency constraints, image border priors, and surface smoothness—are incorporated. The approach enables explicit incorporation of domain knowledge during training, offering a principled way to constrain and steer generative models toward desirable data properties with broad applicability.

Abstract

With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.
Paper Structure (20 sections, 2 theorems, 20 equations, 2 figures, 6 tables)

This paper contains 20 sections, 2 theorems, 20 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Score-matching for energy-based models
  5. Exponential-family distributions
  6. A Hybrid Energy-based Model with Inductive Bias
  7. Training the model
  8. Inject inductive bias through the function $\mathbf{T}(\mathbf{x})$
  9. Statistic function for fitting molecule data
  10. Statistic function for fitting handwritten digits
  11. Statistic function for fitting point clouds
  12. Experiments
  13. Fitting molecular data
  14. Fitting data of hand-written digits
  15. Fitting data of small grayscale images
  16. ...and 5 more sections

Key Result

Theorem 1

If the parameter $\mathbold{\eta}$ is at a local maximum of the $l(\mathbold{\theta}, \mathbold{\eta})$ for a fixed $\mathbold{\theta}$, then ${\mathbb{E}_{p_{\mathbold{\theta}, \mathbold{\eta}}}} \left[\mathbf{T}(\mathbf{x})\right] = \frac{1}{N}\sum_{i=1}^{N} \mathbf{T}(\mathbf{x}_i)$.

Figures (2)

  • Figure 1: All pixels outside the yellow bounding box are zero. This piece of prior knowledge is encoded in the statistics in \ref{['eq:statistic-mnist']}
  • Figure 2: The comparison between $\eta$ values respectively for an arbitrary statistic $\sin(\mathbf{1}^\top \mathbf{x})$ and the mask statistic specified by \ref{['eq:statistic-mnist']}. The result indicates that only specially designed statistics are likely to help the model learn.

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem
  • proof