A Deterministic Sampling Method via Maximum Mean Discrepancy Flow with Adaptive Kernel

TL;DR

This work presents EVI-MMD, a deterministic sampling method that minimizes the Maximum Mean Discrepancy (MMD) via the energetic variational inference (EVI) framework. By formulating particle dynamics under an energy-dissipation principle and solving with an implicit Euler proximal update, the method ensures monotone improvement of a discretized free-energy, while an adaptive Gaussian kernel bandwidth $h_n = a/n^c + b$ balances exploration and exploitation. The approach applies to both fully specified targets and two-sample problems, with cross-term computations efficiently handled through Gaussian sampling or direct training-data terms. Numerical results across toy setups, high-dimensional Gaussians, and generative modeling tasks demonstrate competitive performance against established ParVI methods, highlighting robustness, simplicity, and practical impact for sampling and generative modeling.

Abstract

We propose a novel deterministic sampling method to approximate a target distribution $ρ^*$ by minimizing the kernel discrepancy, also known as the Maximum Mean Discrepancy (MMD). By employing the general \emph{energetic variational inference} framework (Wang et al., 2021), we convert the problem of minimizing MMD to solving a dynamic ODE system of the particles. We adopt the implicit Euler numerical scheme to solve the ODE systems. This leads to a proximal minimization problem in each iteration of updating the particles, which can be solved by optimization algorithms such as L-BFGS. The proposed method is named EVI-MMD. To overcome the long-existing issue of bandwidth selection of the Gaussian kernel, we propose a novel way to specify the bandwidth dynamically. Through comprehensive numerical studies, we have shown the proposed adaptive bandwidth significantly improves the EVI-MMD. We use the EVI-MMD algorithm to solve two types of sampling problems. In the first type, the target distribution is given by a fully specified density function. The second type is a "two-sample problem", where only training data are available. The EVI-MMD method is used as a generative learning model to generate new samples that follow the same distribution as the training data. With the recommended settings of the tuning parameters, we show that the proposed EVI-MMD method outperforms some existing methods for both types of problems.

Figures (10)

• Figure 1: Left: decreasing MMD$^2$ curves with respect to $\theta\in [0,2]$ using different $h$. Right: from the first to the second row and from left to right, for $\theta=0, 1, 1.5, 1.75, 1.9, 2$, the red dots are samples $\{\bm x_{i}^{\theta}\}_{i=1}^N$ and are plotted on the contour plot of the target distribution.
• Figure 2: From top to bottom, each column of sub-figures show the particles by the EVI-MMD algorithm at $n=100$, $n=500$, and $n=1000$ iterations. The target density function is plotted as the contour in the background.
• Figure 3: From left to right, the three sub-figures show the decreasing MMD$^2$ of four methods for the three toy examples with the target distributi:ons star-shaped five-component Gaussian mixture distribution, eight-component Gaussian mixture distribution, and wave-shaped distribution.
• Figure 4: The MMD$^2$ criterion (Gaussian kernel with $h=5$) (left) and energy distance criterion (right) of the particles returned by three methods: blue curve for Support-Point, orange curve for EVI-MMD, and green curve for EVI-Energy Distance.
• Figure 5: Visual comparison between 100 training data randomly chosen from the original training data (left column), new images generated by EVI-MMD (middle column), and EVI-Energy-Distance (right column).