Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy-Guided Sampling of Flat Modes in Discrete Spaces

Pinaki Mohanty, Riddhiman Bhattacharya, Ruqi Zhang

TL;DR

This work addresses the challenge of sampling from flat modes in discrete spaces by introducing Entropic Discrete Langevin Proposal (EDLP), which couples a discrete state with a continuous auxiliary variable through a local-entropy based joint target. The method reuses a smoothed energy $U_{\eta}$ and a joint distribution $p(\widetilde{\bm{\theta}}) \propto \exp\{U(\bm{\theta})-\tfrac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_a\|^2\}$ to steer discrete proposals toward flat regions, yielding EDULA (unadjusted) and EDMALA (MH-adjusted) with a scalable Cat(Softmax) factorization. The authors provide non-asymptotic convergence guarantees for both EDULA and EDMALA under standard smoothness/strong-concavity assumptions, and demonstrate through experiments on Bernoulli distributions, RBMs, TSP, and binary Bayesian neural networks that EDLP discovers and samples from flat basins more effectively than existing baselines. The approach offers a principled way to bias discrete sampling toward robust configurations, with practical implications for combinatorial optimization and discrete generative modeling. The work also includes extensive hyperparameter analyses and provides code to support reproducibility and broader adoption in the community.

Abstract

Sampling from flat modes in discrete spaces is a crucial yet underexplored problem. Flat modes represent robust solutions and have broad applications in combinatorial optimization and discrete generative modeling. However, existing sampling algorithms often overlook the mode volume and struggle to capture flat modes effectively. To address this limitation, we propose \emph{Entropic Discrete Langevin Proposal} (EDLP), which incorporates local entropy into the sampling process through a continuous auxiliary variable under a joint distribution. The local entropy term guides the discrete sampler toward flat modes with a small overhead. We provide non-asymptotic convergence guarantees for EDLP in locally log-concave discrete distributions. Empirically, our method consistently outperforms traditional approaches across tasks that require sampling from flat basins, including Bernoulli distribution, restricted Boltzmann machines, combinatorial optimization, and binary neural networks.

Entropy-Guided Sampling of Flat Modes in Discrete Spaces

TL;DR

This work addresses the challenge of sampling from flat modes in discrete spaces by introducing Entropic Discrete Langevin Proposal (EDLP), which couples a discrete state with a continuous auxiliary variable through a local-entropy based joint target. The method reuses a smoothed energy and a joint distribution to steer discrete proposals toward flat regions, yielding EDULA (unadjusted) and EDMALA (MH-adjusted) with a scalable Cat(Softmax) factorization. The authors provide non-asymptotic convergence guarantees for both EDULA and EDMALA under standard smoothness/strong-concavity assumptions, and demonstrate through experiments on Bernoulli distributions, RBMs, TSP, and binary Bayesian neural networks that EDLP discovers and samples from flat basins more effectively than existing baselines. The approach offers a principled way to bias discrete sampling toward robust configurations, with practical implications for combinatorial optimization and discrete generative modeling. The work also includes extensive hyperparameter analyses and provides code to support reproducibility and broader adoption in the community.

Abstract

Sampling from flat modes in discrete spaces is a crucial yet underexplored problem. Flat modes represent robust solutions and have broad applications in combinatorial optimization and discrete generative modeling. However, existing sampling algorithms often overlook the mode volume and struggle to capture flat modes effectively. To address this limitation, we propose \emph{Entropic Discrete Langevin Proposal} (EDLP), which incorporates local entropy into the sampling process through a continuous auxiliary variable under a joint distribution. The local entropy term guides the discrete sampler toward flat modes with a small overhead. We provide non-asymptotic convergence guarantees for EDLP in locally log-concave discrete distributions. Empirically, our method consistently outperforms traditional approaches across tasks that require sampling from flat basins, including Bernoulli distribution, restricted Boltzmann machines, combinatorial optimization, and binary neural networks.
Paper Structure (32 sections, 13 theorems, 90 equations, 15 figures, 4 tables, 1 algorithm)

This paper contains 32 sections, 13 theorems, 90 equations, 15 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Entropic Discrete Langevin Proposal
  5. Target Joint Distribution: Coupling Mechanism
  6. Sampling Algorithm: Local Entropy Guidance in Discrete Langevin Proposals
  7. Theoretical Analysis
  8. Convergence Analysis for EDULA
  9. Convergence Analysis for EDMALA
  10. Experiments
  11. Motivational Synthetic Example
  12. Sampling for Traveling Salesman Problems
  13. Sampling From Restricted Boltzmann Machines
  14. Binary Bayesian Neural Networks
  15. Discussion
  16. ...and 17 more sections

Key Result

Lemma 4.1

Given $\widetilde{\bm{\theta}} = [\bm{\theta}^T, \bm{\theta}_a^T]^T \in \bm{\Theta}\times \mathbb{R}^{d}$, the joint distribution p($\widetilde{\bm{\theta}}$) is: By construction, the marginal distributions of $\bm{\theta}$ and $\bm{\theta}_a$ are the original distribution $p(\bm{\theta})$ and the smoothed distribution $p(\bm{\theta}_a)$ (Eq. eq:theta_a_posterior).

Figures (15)

  • Figure 1: Cost landscape visualization on Traveling Salesman Problem (TSP). Flat modes imply robust solutions under budget, whereas sharp modes are highly sensitive to small changes, leading to abrupt cost increases.
  • Figure 2: Overlay Heatmaps for EDULA, EDMALA, DULA, and DMALA.
  • Figure 3: Performance of various samplers on TSP.
  • Figure 4: $p(\bm{\bm{\theta}_a})$ for $\bm{\theta} \sim \textit{Bernoulli}(0.5)$
  • Figure 5: Diagnostics for EDLP
  • ...and 10 more figures

Theorems & Definitions (13)

  • Lemma 4.1
  • Proposition 5.4
  • Theorem 5.5
  • Theorem 5.6
  • Theorem D.1
  • Theorem D.2
  • Lemma D.3
  • Theorem D.4
  • Theorem D.5
  • Lemma D.6
  • ...and 3 more