Efficient Stochastic Optimisation via Sequential Monte Carlo

TL;DR

This work tackles optimisation where the gradient is an expectation under an intractable distribution, causing costly inner sampling. It introduces SOSMC, a flexible SMC-based framework that replaces inner MCMC loops with sequential particle methods to estimate gradients, backed by a Feynman–Kac identity and ESS-based analysis. The paper proves convergence in an idealised setting and discusses the impact of particle approximations, while demonstrating practical gains through extensive experiments on reward-tuning of EBMs, including Langevin processes, 2D EBMs, and MNIST. The results show SOSMC achieving faster convergence and more accurate gradient estimates than competing approaches, highlighting its potential for scalable, efficient optimisation in models with intractable gradients. This framework unifies and extends existing SMC-based and EM-like strategies, offering a practical route to accelerate training and tuning in energy-based and generative modelling contexts.

Abstract

The problem of optimising functions with intractable gradients frequently arise in machine learning and statistics, ranging from maximum marginal likelihood estimation procedures to fine-tuning of generative models. Stochastic approximation methods for this class of problems typically require inner sampling loops to obtain (biased) stochastic gradient estimates, which rapidly becomes computationally expensive. In this work, we develop sequential Monte Carlo (SMC) samplers for optimisation of functions with intractable gradients. Our approach replaces expensive inner sampling methods with efficient SMC approximations, which can result in significant computational gains. We establish convergence results for the basic recursions defined by our methodology which SMC samplers approximate. We demonstrate the effectiveness of our approach on the reward-tuning of energy-based models within various settings.

Figures (18)

• Figure 1: Wall-clock convergence of mean reward (left) and NLL (right), across $10$ runs, for ImpDiff, SOUL, and SOSMC under different $\mathrm{OPT}$ for (a)$V_{\mathrm{dual}}$ & $R_{\mathrm{smooth}}$; (b)$V_{\mathrm{sparse}}$ & $R_{\mathrm{hard}}$.
• Figure 2: Tuning trajectories for ImpDiff and SOSMC-ULA in the $(\widehat{R}_{\mathrm{fresh}},\widehat{\mathrm{KL}})$ plane, for $\beta_{\mathrm{KL}} = 0.25$. Contours denote level sets of $\ell$, with SOSMC-ULA to the right of (i.e. better than) the best ImpDiff solution achieved during tuning (red).
• Figure 3: Terminal density snapshots of $\pi_{\theta_{K}}$, using identical initialisation and shared noise, across increasing $\beta_{\mathrm{KL}}$, for $R_{\mathrm{lower}}$.
• Figure 4: Density snapshots along the sampling evolution of $\pi_{\theta_{K}}$ with half-plane reward for both ImpDiff (top) and SOSMC-ULA (bottom), for $\beta_{\mathrm{KL}} = 0.25$, with identical initialisation and shared noise. Mass concentrates in the $R_{\mathrm{lower}}$ region significantly faster for SOSMC-ULA.
• Figure 5: Example sampling trajectories, with step index, under the pre-training sampler kernel, for the pre-trained model (top), and $\pi_{\theta_{K}}$ for both ImpDiff (middle) and SOSMC-ULA (bottom). In (a) the reward favours mass in the lower half-plane, whilst in (b) reward favours brighter images, as outlined in Appendix \ref{['appdx:mnist-tuning-details']}.

Theorems & Definitions (23)

• Theorem 1: del2004feynman
• proof
• Remark 1
• Remark 2
• Lemma 1: Exact gradient recovery
• proof
• Proposition 2: Convergence of the idealised scheme.
• proof
• Remark 3
• Proposition 3