Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concrete Score Matching: Generalized Score Matching for Discrete Data

Chenlin Meng, Kristy Choi, Jiaming Song, Stefano Ermon

TL;DR

The paper introduces the Concrete score as a discrete analogue of the Stein score by leveraging a neighborhood-based surrogate for gradient information in discrete spaces. It then defines Concrete Score Matching (CSM) to learn these scores from data, with efficient Monte Carlo training and a denoising variant (D-CSM) that mirrors denoising score matching in continuous domains. The approach connects to continuous score methods through a limiting relationship and demonstrates strong performance on synthetic, tabular, and high-dimensional discrete data, including MNIST-style images, using MH-based sampling. Overall, CSM provides a scalable framework for density estimation and sampling in discrete domains, expanding the applicability of score-based modeling to structured discrete data.

Abstract

Representing probability distributions by the gradient of their density functions has proven effective in modeling a wide range of continuous data modalities. However, this representation is not applicable in discrete domains where the gradient is undefined. To this end, we propose an analogous score function called the "Concrete score", a generalization of the (Stein) score for discrete settings. Given a predefined neighborhood structure, the Concrete score of any input is defined by the rate of change of the probabilities with respect to local directional changes of the input. This formulation allows us to recover the (Stein) score in continuous domains when measuring such changes by the Euclidean distance, while using the Manhattan distance leads to our novel score function in discrete domains. Finally, we introduce a new framework to learn such scores from samples called Concrete Score Matching (CSM), and propose an efficient training objective to scale our approach to high dimensions. Empirically, we demonstrate the efficacy of CSM on density estimation tasks on a mixture of synthetic, tabular, and high-dimensional image datasets, and demonstrate that it performs favorably relative to existing baselines for modeling discrete data.

Concrete Score Matching: Generalized Score Matching for Discrete Data

TL;DR

The paper introduces the Concrete score as a discrete analogue of the Stein score by leveraging a neighborhood-based surrogate for gradient information in discrete spaces. It then defines Concrete Score Matching (CSM) to learn these scores from data, with efficient Monte Carlo training and a denoising variant (D-CSM) that mirrors denoising score matching in continuous domains. The approach connects to continuous score methods through a limiting relationship and demonstrates strong performance on synthetic, tabular, and high-dimensional discrete data, including MNIST-style images, using MH-based sampling. Overall, CSM provides a scalable framework for density estimation and sampling in discrete domains, expanding the applicability of score-based modeling to structured discrete data.

Abstract

Representing probability distributions by the gradient of their density functions has proven effective in modeling a wide range of continuous data modalities. However, this representation is not applicable in discrete domains where the gradient is undefined. To this end, we propose an analogous score function called the "Concrete score", a generalization of the (Stein) score for discrete settings. Given a predefined neighborhood structure, the Concrete score of any input is defined by the rate of change of the probabilities with respect to local directional changes of the input. This formulation allows us to recover the (Stein) score in continuous domains when measuring such changes by the Euclidean distance, while using the Manhattan distance leads to our novel score function in discrete domains. Finally, we introduce a new framework to learn such scores from samples called Concrete Score Matching (CSM), and propose an efficient training objective to scale our approach to high dimensions. Empirically, we demonstrate the efficacy of CSM on density estimation tasks on a mixture of synthetic, tabular, and high-dimensional image datasets, and demonstrate that it performs favorably relative to existing baselines for modeling discrete data.
Paper Structure (41 sections, 11 theorems, 37 equations, 8 figures, 2 tables, 2 algorithms)

This paper contains 41 sections, 11 theorems, 37 equations, 8 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Concrete Score
  4. Constructing Surrogate Gradients for Discrete Data
  5. Connection to Stein Scores and Existing Score Matching Techniques
  6. Inference with Concrete Scores
  7. Learning Concrete Scores with Concrete Score Matching
  8. The Concrete Score Matching Objective
  9. Efficient Training via Monte Carlo
  10. Denoising Concrete Score Matching
  11. Experimental Results
  12. Baselines
  13. 1-D Discrete Data
  14. Sampling with Toy 2-D Multi-Class Datasets
  15. Likelihood Evaluation on Discrete Tabular Data
  16. ...and 26 more sections

Key Result

Theorem 1

Let $p_{\rm{data}}(\mathbf{x})$ be a (discrete) data distribution. Denote ${\bm{c}}_{p_{\rm{data}}}(\mathbf{x}; \mathcal{N})$ as the Concrete score of $p_{\rm{data}}({\mathbf{x}})$ with neighboring structure $\mathcal{N}$, and ${\bm{c}}_\theta(\mathbf{x}; \mathcal{N})$ the Concrete score for a dist

Figures (8)

  • Figure 1: Examples of common neighborhood structures and their corresponding (connected) neighborhood-induced graphs, where the arrows point to the neighbors of a given node. Note that the neighborhood structure is directed.
  • Figure 2: Sampling results from a toy 1-D discrete dataset. CSM recovers the true data distribution $p_{\rm{data}}({\mathbf{x}})$ using Metropolis-Hastings (left). The Concrete score can also be used to recover the Stein score of the triangular noise-perturbed data distribution, allowing for sampling with Langevin dynamics. Such samples can be denoised to recover the original (clean) data distribution (right).
  • Figure 3: Sampling results on toy 2-D benchmark datasets. We find that CSM produces the highest quality samples across all 3 datasets relative to all baselines.
  • Figure 4: Image samples from CSM on the binarized MNIST dataset. We observe that CSM is able generate high-quality samples on MNIST using Metropolis-Hastings.
  • Figure 5: Examples of neighborhood structures and their corresponding log-likelihood values (higher is better) when trained with Concrete-SM.
  • ...and 3 more figures

Theorems & Definitions (24)

  • Definition 1: Neighborhood-induced graph
  • Example 1
  • Definition 2: Concrete score
  • Theorem 1: Completeness
  • proof : Proof sketch.
  • Proposition 1
  • Theorem 2: Consistency
  • Theorem 3
  • Theorem 4: Denoising Concrete Score Matching
  • Theorem 4: Completeness
  • ...and 14 more