Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consensus-Based Optimization with Truncated Noise

Massimo Fornasier, Peter Richtárik, Konstantin Riedl, Lukang Sun

TL;DR

This paper rigorously proves convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialization, allowing in particular for wider flexibility in choosing the noise parameter of the method as confirmed experimentally.

Abstract

Consensus-based optimization (CBO) is a versatile multi-particle metaheuristic optimization method suitable for performing nonconvex and nonsmooth global optimizations in high dimensions. It has proven effective in various applications while at the same time being amenable to a theoretical convergence analysis. In this paper, we explore a variant of CBO, which incorporates truncated noise in order to enhance the well-behavedness of the statistics of the law of the dynamics. By introducing this additional truncation in the noise term of the CBO dynamics, we achieve that, in contrast to the original version, higher moments of the law of the particle system can be effectively bounded. As a result, our proposed variant exhibits enhanced convergence performance, allowing in particular for wider flexibility in choosing the noise parameter of the method as we confirm experimentally. By analyzing the time-evolution of the Wasserstein-$2$ distance between the empirical measure of the interacting particle system and the global minimizer of the objective function, we rigorously prove convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialization. Numerical evidences demonstrate the benefit of truncating the noise in CBO.

Consensus-Based Optimization with Truncated Noise

TL;DR

This paper rigorously proves convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialization, allowing in particular for wider flexibility in choosing the noise parameter of the method as confirmed experimentally.

Abstract

Consensus-based optimization (CBO) is a versatile multi-particle metaheuristic optimization method suitable for performing nonconvex and nonsmooth global optimizations in high dimensions. It has proven effective in various applications while at the same time being amenable to a theoretical convergence analysis. In this paper, we explore a variant of CBO, which incorporates truncated noise in order to enhance the well-behavedness of the statistics of the law of the dynamics. By introducing this additional truncation in the noise term of the CBO dynamics, we achieve that, in contrast to the original version, higher moments of the law of the particle system can be effectively bounded. As a result, our proposed variant exhibits enhanced convergence performance, allowing in particular for wider flexibility in choosing the noise parameter of the method as we confirm experimentally. By analyzing the time-evolution of the Wasserstein- distance between the empirical measure of the interacting particle system and the global minimizer of the objective function, we rigorously prove convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialization. Numerical evidences demonstrate the benefit of truncating the noise in CBO.
Paper Structure (16 sections, 10 theorems, 101 equations, 1 figure, 5 tables)

This paper contains 16 sections, 10 theorems, 101 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Motivation for using truncated noise.
  3. Contributions.
  4. Organization
  5. Notation
  6. Global Convergence of CBO with Truncated Noise
  7. Main Result
  8. Proof Details for Section \ref{['sec:main_result']}
  9. Well-Posedness of Equations \ref{['eq:17']} and \ref{['eq:5']}
  10. Proof Details for Theorem \ref{['thm:main1']}
  11. Upper Bound for the Second Term in \ref{['eq:error_decomposition']}
  12. Upper Bound for the Third Term in \ref{['eq:error_decomposition']}
  13. Numerical Experiments
  14. Isotropic Case.
  15. Anisotropic Case.
  16. ...and 1 more sections

Key Result

Theorem 3

Let $f \in {\cal C}(\mathbb{R}^d)$ satisfy asp:11, asp:22 and asp:33. Moreover, let $\rho_0\in{\cal P}_4(\mathbb{R}^d)$ with $v^*\in\operatorname{supp}(\rho_0)$. Let $V^i_{ 0,\Delta t }$ be sampled i.i.d. from $\rho_0$ and denote by $((V^i_{ {k,\Delta t} })_{k=1,\dots,K})_{i=1,\dots,N}$ the iteratio and let $K \in \mathbb{N}$ and $\Delta t$ satisfy ${{K\Delta t}}=T^*$. Moreover, let $R\in (\!\left

Figures (1)

  • Figure 1: A comparison of the success probabilities of isotropic CBO with (left phase diagrams) and without (right separate columns) truncated noise for different values of the truncation parameter $M$ and the noise level $\sigma$. (Note that standard CBO as investigated in pinnau2017consensuscarrillo2018analyticalfornasier2021consensus is retrieved when choosing $M=\infty$, $R=\infty$ and $v_b=0$ in \ref{['eq:17']}). In both settings (a) and (b) the depicted success probabilities are averaged over $100$ runs and the implemented scheme is given by an Euler-Maruyama discretization of Equation \ref{['eq:5']} with time horizon $T=50$, discrete time step size $\Delta t=0.01$, $R=\infty$, $v_b=0$, $\alpha=10^5$ and $\lambda=1$. We use $N=100$ particles, which are initialized according to $\rho_0={\cal N}((1,\dots,1),2000)$. In both figures we plot the success probability of standard CBO (right separate column) and the CBO variant with truncated noise (left phase transition diagram) for different values of the truncation parameter $M$ and the noise level $\sigma$, when optimizing the Ackley ((a)) and Rastrigin ((b)) function, respectively. We observe that truncating the noise term (by decreasing $M$) consistently allows for a wider flexibility when choosing the noise level $\sigma$ and thus increasing the likelihood of successfully locating the global minimizer.

Theorems & Definitions (26)

  • Remark 1: Sub-Gaussianity of truncated CBO
  • Definition 2: Assumptions
  • Theorem 3
  • Remark 4
  • Remark 5
  • Remark 6
  • Proposition 7
  • proof
  • Lemma 8
  • proof
  • ...and 16 more