Electrostatics-based particle sampling and approximate inference

TL;DR

The paper addresses the challenge of sampling and inference for complex target densities that may be difficult to access with gradients. It introduces EParVI, a deterministic, gradient-free interacting particle system inspired by electrostatics, where negative charges migrate under repulsion and attraction to fixed positives whose magnitudes encode the target density $p(oldsymbol{x})$, driving the particle distribution toward the target. The authors formalize repulsive and attractive forces, propose three discrete-time update rules (Euler, Verlet, damped-Verlet), and demonstrate competitive performance on low-dimensional densities, Neal's funnel, Bayesian logistic regression, and Lotka–Volterra parameter inference, while providing discussion on complexity, diagnostics, and extensions. They argue that this physics-based, non-gradient-based approach offers a simple, non-parametric, extensible alternative to traditional MCMC/VI methods, with potential for broader probabilistic machine learning applications, albeit with notable dimensionality challenges and opportunities for efficiency gains.

Abstract

A new particle-based sampling and approximate inference method, based on electrostatics and Newton mechanics principles, is introduced with theoretical ground, algorithm design and experimental validation. This method simulates an interacting particle system (IPS) where particles, i.e. the freely-moving negative charges and spatially-fixed positive charges with magnitudes proportional to the target distribution, interact with each other via attraction and repulsion induced by the resulting electric fields described by Poisson's equation. The IPS evolves towards a steady-state where the distribution of negative charges conforms to the target distribution. This physics-inspired method offers deterministic, gradient-free sampling and inference, achieving comparable performance as other particle-based and MCMC methods in benchmark tasks of inferring complex densities, Bayesian logistic regression and dynamical system identification. A discrete-time, discrete-space algorithmic design, readily extendable to continuous time and space, is provided for usage in more general inference problems occurring in probabilistic machine learning scenarios such as Bayesian inference, generative modelling, and beyond.

Figures (17)

• Figure 1: Illustrative diagram of particle distributions. Red darkness indicates the density value of two-Gaussian mixture, arrows denote example forces. Positive charges are fixed at grid points with magnitude proportional to density value, negative charges can move freely.
• Figure 2: Two toy 2D Gaussian densities. Upper: unimodal, lower: bimodal. Left to right: iteration 0, 5, 60 and selected particle trajectories (dot: start, square: end). Pink contours are ground truth density; blue dots denote negative charges, red dots denote positive charges. See Appendix.\ref{['app:toy_examples']} for details.
• Figure 3: Three other toy 2D densities. Left to right: iteration 0, 40, 60. See Appendix.\ref{['app:toy_examples']} for details.
• Figure 4: Inference of the Neal's funnel using six methods. Upper left to right: LMC, SVGD, EVI, EParVI, and MH samples. Lower left to right: corresponding $MMD^2$, $NLL$ values over runtime and HMC samples. MH and HMC take seconds to run. See Appendix.\ref{['app:neals_funnel_implementation_details']} for details.
• Figure 5: Inference of BLR model. Upper left to right: HMC and SVGD samples. Lower left to right: EVI and EParVI samples. MH and HMC take seconds to run. See Appendix.\ref{['app:BLR']} for details.