Smoothing Iterative Consensus-based Optimization Algorithm for Nonsmooth Nonconvex Optimization Problems with Global Optimality
Jiazhen Wei, Wei Bian
TL;DR
The paper addresses the global minimization of a general unconstrained nonsmooth nonconvex objective $f:\mathbb{R}^d\to\mathbb{R}_+$ and proposes a smoothing-based workaround. It introduces the SICBO algorithm, a discrete, smoothing-augmented consensus-based optimization method that avoids mean-field limits and is applicable to nonsmooth and possibly non-Lipschitz objectives via a convolution-based smoothing $\tilde{f}(x,\mu)$. The authors prove that the particle system exhibits exponential convergence to a common consensus point $x_\infty$ (in expectation and almost surely) and provide an error bound ${\rm ess}\inf_{\omega} f(x_\infty(\omega)) \le f(x^*) + E(\beta)$ with $E(\beta)\to0$ as $\beta\to\infty$ under suitable conditions. Numerical experiments demonstrate effective global optimization performance and competitiveness with stochastic subgradient methods, including applications to deep neural network training.
Abstract
In this paper, we focus on finding the global minimizer of a general unconstrained nonsmooth nonconvex optimization problem. Taking advantage of the smoothing method and the consensus-based optimization (CBO) method, we propose a novel smoothing iterative consensus-based optimization (SICBO) algorithm. First, we prove that the solution process of the proposed algorithm here exponentially converges to a common stochastic consensus point almost surely. Second, we establish a detailed theoretical analysis to ensure the small enough error between the objective function value at the consensus point and the optimal function value, to the best of our knowledge, which provides the first theoretical guarantee to the global optimality of the proposed algorithm for nonconvex optimization problems. Moreover, unlike the previously introduced CBO methods, the theoretical results are valid for the cases that the objective function is nonsmooth, nonconvex and perhaps non-Lipschitz continuous. Finally, several numerical examples are performed to illustrate the effectiveness of our proposed algorithm for solving the global minimizer of the nonsmooth and nonconvex optimization problems.