On scalable and efficient training of diffusion samplers

TL;DR

This work tackles training diffusion samplers for unnormalized densities in data-poor settings by coupling a gradient-guided MCMC Searcher with a diffusion-based Learner in a two-stage SGDS framework. The key idea is to use off-policy trajectories from a Searcher, augmented with a novelty-based exploration reward, to train the Learner via trajectory balance on a mixture of searcher- and self-generated trajectories, while periodically reinitializing to counter primacy bias. SGDS demonstrates stronger sample efficiency and scalability on high-dimensional benchmarks (Manywell, LJ) and real-world molecular conformer generation (Alanine peptides) by expanding mode coverage and improving energy-quality samples. The approach leverages RND-based intrinsic rewards to steer exploration toward underexplored regions, enabling robust performance across tasks and making diffusion samplers more practical for complex energy landscapes. Overall, SGDS provides a principled, scalable path to integrate classical sampling with neural amortization for high-dimensional diffusion-based inference.

Abstract

We address the challenge of training diffusion models to sample from unnormalized energy distributions in the absence of data, the so-called diffusion samplers. Although these approaches have shown promise, they struggle to scale in more demanding scenarios where energy evaluations are expensive and the sampling space is high-dimensional. To address this limitation, we propose a scalable and sample-efficient framework that properly harmonizes the powerful classical sampling method and the diffusion sampler. Specifically, we utilize Monte Carlo Markov chain (MCMC) samplers with a novelty-based auxiliary energy as a Searcher to collect off-policy samples, using an auxiliary energy function to compensate for exploring modes the diffusion sampler rarely visits. These off-policy samples are then combined with on-policy data to train the diffusion sampler, thereby expanding its coverage of the energy landscape. Furthermore, we identify primacy bias, i.e., the preference of samplers for early experience during training, as the main cause of mode collapse during training, and introduce a periodic re-initialization trick to resolve this issue. Our method significantly improves sample efficiency on standard benchmarks for diffusion samplers and also excels at higher-dimensional problems and real-world molecular conformer generation.

Figures (7)

• Figure 1: Mode coverage comparison using 2D projections of 10,000 samples on Manywell-128.
• Figure 2: Trade-off between EUBO–ELBO gap and energy calls in Manywell-128 (left) and LJ-55 (middle). The results of ablation study on Manywell-128 (right) show the performance of AIS using the same total energy calls with MLE amortizing, taking 2 rounds with fine-tuning instead of re-initialization, and using the Searcher with no RND rewards. All methods use 20M energy calls.
• Figure 3: Histograms for LJ-13/55 energy densities and LJ-55 interatomic distances.
• Figure 4: Qualitative results of methods in three peptides. (a) Ramachandran plot of Alanine Dipeptide with two backbone torsion angles $(\phi,\psi)$ and (b) 3D visualization of generated conformations.
• Figure 5: Mode coverage comparison on 40GMM.