Swarm-based optimization with jumps: a kinetic BGK framework and convergence analysis

TL;DR

This work introduces a novel swarm-based optimization method in which particle velocities evolve via stochastic jumps, and it casts the dynamics in a kinetic BGK framework that accommodates general noise distributions, including heavy-tailed ones. A mean-field limit is derived and shown to connect to Consensus-Based Optimization (CBO) through a diffusion scaling, while propagation of chaos and convergence to global minimizers are established for bounded domains under Gaussian noise. Theoretical results are complemented by extensive numerical experiments on standard nonconvex benchmarks, demonstrating robust performance and revealing critical roles for the diffusion level and jump frequency. Overall, the paper provides a rigorous link between jump-enabled swarm dynamics and kinetic PDE methods, offering guidance for parameter scaling and noise design with practical implications for global optimization.

Abstract

Metaheuristic algorithms are powerful tools for global optimization, particularly for non-convex and non-differentiable problems where exact methods are often impractical. Particle-based optimization methods, inspired by swarm intelligence principles, have shown effectiveness due to their ability to balance exploration and exploitation within the search space. In this work, we introduce a novel particle-based optimization algorithm where velocities are updated via random jumps, a strategy commonly used to enhance stochastic exploration. We formalize this approach by describing the dynamics through a kinetic modelling of BGK type, offering a unified framework that accommodates general noise distributions, including heavy-tailed ones like Cauchy. Under suitable parameter scaling, the model reduces to the Consensus-Based Optimization (CBO) dynamics. For non-degenerate Gaussian noise in bounded domains, we prove propagation of chaos and convergence towards minimizers. Numerical results on benchmark problems validate the approach and highlight its connection to CBO.

Figures (3)

• Figure 1: Variations in diffusion coefficient $\sigma$ using Gaussian (top) and Cauchy (bottom) noise. Parameters are set to $N=200, \Delta t=0.1, \lambda=1, \nu=1, \alpha=10^5$, with total iteration step $k_T=10^3$. For visualisation purposes, values exceeding $10^7$ were excluded. We note that, although the mean $l^\infty$ error is small, the fitness values can still become large due to the definition of the Rosenbrock and XSY random function.
• Figure 2: Density plot snapshots in the diffusion limit for the Rastrigin function in $d=1$, shown for rescaled jump PSO \ref{['alg_BGK2']} with $\varepsilon = 1$ and $\varepsilon = 0.1$, alongside the CBO dynamics. The parameters are set to $N=1000$, $\tilde{\sigma}=1$ (for CBO), $\lambda=1$, $\Delta t = 0.1$, and $\alpha = 10^5$, with a total of $k_T= 10^3$ iteration steps. The diffusion constants for the scaled algorithm are chosen according to the scaling law \ref{['eq:sigma-scale']}. Particles are visualised in the rescaled domain $[-1,1]$ and the density function is obtained via kernel density reconstruction. For reference, the objective function (Rastrigin) is plotted in the background with a separate axis on the right.
• Figure 3: Comparison with CBO via scaling limit. Tested on a range of $\tilde{\sigma}$ values from $0$ to $5$. For each $\varepsilon$ values, we plug in $\sigma=\tilde{\sigma}\varepsilon/\sqrt{\Delta t}$ in our algorithm. Ackley (left) and Rastrigin (right) functions are used in $d=20$, parameters are set to $N=200, \Delta t=0.1, \lambda=1, \nu=1, \alpha=10^5$, with total iteration step $k_T=10^3$.

Theorems & Definitions (27)

• Remark 2.1
• Lemma 2.1
• proof
• Remark 2.2
• Remark 2.3
• Remark 2.4: Fluid limit
• Theorem 3.1
• Corollary 3.1
• Lemma 3.1
• proof