Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

Massimo Fornasier, Timo Klock, Konstantin Riedl

TL;DR

The proof technique reveals that CBO performs a convexification of the optimization problem as the number of agents goes to infinity, thus providing an insight into the internal CBO mechanisms responsible for the success of the method.

Abstract

In this paper we study anisotropic consensus-based optimization (CBO), a multi-agent metaheuristic derivative-free optimization method capable of globally minimizing nonconvex and nonsmooth functions in high dimensions. CBO is based on stochastic swarm intelligence, and inspired by consensus dynamics and opinion formation. Compared to other metaheuristic algorithms like particle swarm optimization, CBO is of a simpler nature and therefore more amenable to theoretical analysis. By adapting a recently established proof technique, we show that anisotropic CBO converges globally with a dimension-independent rate for a rich class of objective functions under minimal assumptions on the initialization of the method. Moreover, the proof technique reveals that CBO performs a convexification of the optimization problem as the number of agents goes to infinity, thus providing an insight into the internal CBO mechanisms responsible for the success of the method. To motivate anisotropic CBO from a practical perspective, we further test the method on a complicated high-dimensional benchmark problem, which is well understood in the machine learning literature.

Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

TL;DR

The proof technique reveals that CBO performs a convexification of the optimization problem as the number of agents goes to infinity, thus providing an insight into the internal CBO mechanisms responsible for the success of the method.

Abstract

In this paper we study anisotropic consensus-based optimization (CBO), a multi-agent metaheuristic derivative-free optimization method capable of globally minimizing nonconvex and nonsmooth functions in high dimensions. CBO is based on stochastic swarm intelligence, and inspired by consensus dynamics and opinion formation. Compared to other metaheuristic algorithms like particle swarm optimization, CBO is of a simpler nature and therefore more amenable to theoretical analysis. By adapting a recently established proof technique, we show that anisotropic CBO converges globally with a dimension-independent rate for a rich class of objective functions under minimal assumptions on the initialization of the method. Moreover, the proof technique reveals that CBO performs a convexification of the optimization problem as the number of agents goes to infinity, thus providing an insight into the internal CBO mechanisms responsible for the success of the method. To motivate anisotropic CBO from a practical perspective, we further test the method on a complicated high-dimensional benchmark problem, which is well understood in the machine learning literature.
Paper Structure (17 sections, 6 theorems, 49 equations, 4 figures)

This paper contains 17 sections, 6 theorems, 49 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Prior Arts.
  3. Contribution.
  4. Organization.
  5. Notation.
  6. Global Convergence in Mean-Field Law
  7. Definition of Weak Solutions and Well-Posedness
  8. Main Results
  9. Proof of Theorem \ref{['thm:global_convergence_main_anisotropic']}
  10. Proof Sketch
  11. Evolution of the Mean-Field Limit
  12. Quantitative Laplace Principle
  13. A Lower Bound for the Probability Mass around $v^*$
  14. Proof of Theorem \ref{['thm:global_convergence_main_anisotropic']}
  15. A Machine Learning Example
  16. ...and 2 more sections

Key Result

theorem 1

Let $T > 0$, $\rho_0 \in {\cal P}_4(\mathbb{R}^d)$ and consider ${\cal E} : \mathbb{R}^d\rightarrow \mathbb{R}$ with $\underbar {\cal E}:={\cal E}(v^*) > -\infty$, which, for some constants $C_1,C_2 > 0$, satisfies If in addition, either $\sup_{v \in \mathbb{R}^d}{\cal E}(v) < \infty$, or, for some $C_3,C_4 > 0$, ${\cal E}$ satisfies then there exists a law $\rho \in {\cal C}([0,T], {\cal P}_4(\

Figures (4)

  • Figure 1: A demonstration of the benefit of using anisotropic diffusion in CBO. For the Rastrigin function ${\cal E}(v)\!=\!\sum_{k=1}^d \!v_k^2\!+\!\frac{5}{2}(1\!-\!\cos(2\pi v_k))$ with $v^* \!=\! 0$ and spurious local minima (see (a)), we evolve the discretized system of isotropic and anisotropic CBO using $N = 320000$ particles, discrete time step size $\Delta t = 0.01$ and $\alpha = 10^{15}$, $\lambda = 1$, and $\sigma = 0.32$ for different dimensions $d\in\{4,8,12,16\}$. We observe in (b) that the convergence rate of the energy functional ${\cal V}(\widehat{\rho}_{t}^N)$ for isotropic CBO (dashed lines) is affected by the ambient dimension $d$, whereas anisotropic CBO (solid lines) converges independently from $d$ with rate $(2\lambda-\sigma^2)$.
  • Figure 2: Visualization of the decomposition of $\Omega_r$ as in \ref{['eq:disjoint_sets']} for different positions of $v_{\alpha}({\rho_t})$ and values of $\sigma$.
  • Figure 3: Architectures of the NNs used in the experiments of Section \ref{['sec:numerics']}.
  • Figure 4: Comparison of the performances of a shallow (dashed lines) and convolutional (solid lines) NN with architectures as described in Figures \ref{['fig:shallowNN']} and \ref{['fig:CNN']}, when trained with a discretized version of the anisotropic CBO dynamics \ref{['eq:dyn_micro']}. Depicted are the accuracies on a test dataset (orange lines) and the values of the objective function ${\cal E}$ (blue lines), which was chosen to be the categorical crossentropy loss on a random sample of the training set of size $10000$.

Theorems & Definitions (17)

  • definition 1
  • theorem 1
  • proof
  • remark 1
  • definition 2: Assumptions
  • theorem 2
  • remark 2
  • lemma 1
  • proof
  • proposition 1
  • ...and 7 more