Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forward-backward splitting under the light of generalized convexity

Konstantinos Oikonomidis, Emanuel Laude, Panagiotis Patrinos

TL;DR

The paper develops a unifying framework for continuous optimization based on generalized convexity ($\Phi$-convexity), extending the classical difference-of-convex approach. By formulating the problem as $F(x)=g(x)-f(x)$ with a coupling $\Phi$, it derives a dual theory ($\Phi$-duality) and a forward-backward algorithm ($\Phi$-DCA) that encompasses many known methods (PGM, BPGM, NGD, tensor methods) as special cases. The authors introduce gap functions and a $\Phi$-DCA-PL inequality to establish convergence and linear rates, along with a $\Phi$-Bregman proximal interpretation for sublinear regimes. They demonstrate that numerous classical results are instances of their framework, and provide sublinear and linear rate analyses under various smoothness and convexity assumptions, including anisotropic and Hölder settings. The framework promises a streamlined convergence analysis and a flexible platform for designing new algorithms within generalized convexity.

Abstract

In this paper we present a unifying framework for continuous optimization methods grounded in the concept of generalized convexity. Utilizing the powerful theory of $Φ$-convexity, we propose a conceptual algorithm that extends the classical difference-of-convex method, encompassing a broad spectrum of optimization algorithms. Relying exclusively on the tools of generalized convexity we develop a gap function analysis that strictly characterizes the decrease of the function values, leading to simplified and unified convergence results. As an outcome of this analysis, we naturally obtain a generalized PL inequality which ensures $q$-linear convergence rates of the proposed method, incorporating various well-established conditions from the existing literature. Moreover we propose a $Φ$-Bregman proximal point interpretation of the scheme that allows us to capture conditions that lead to sublinear rates under convexity.

Forward-backward splitting under the light of generalized convexity

TL;DR

The paper develops a unifying framework for continuous optimization based on generalized convexity (-convexity), extending the classical difference-of-convex approach. By formulating the problem as with a coupling , it derives a dual theory (-duality) and a forward-backward algorithm (-DCA) that encompasses many known methods (PGM, BPGM, NGD, tensor methods) as special cases. The authors introduce gap functions and a -DCA-PL inequality to establish convergence and linear rates, along with a -Bregman proximal interpretation for sublinear regimes. They demonstrate that numerous classical results are instances of their framework, and provide sublinear and linear rate analyses under various smoothness and convexity assumptions, including anisotropic and Hölder settings. The framework promises a streamlined convergence analysis and a flexible platform for designing new algorithms within generalized convexity.

Abstract

In this paper we present a unifying framework for continuous optimization methods grounded in the concept of generalized convexity. Utilizing the powerful theory of -convexity, we propose a conceptual algorithm that extends the classical difference-of-convex method, encompassing a broad spectrum of optimization algorithms. Relying exclusively on the tools of generalized convexity we develop a gap function analysis that strictly characterizes the decrease of the function values, leading to simplified and unified convergence results. As an outcome of this analysis, we naturally obtain a generalized PL inequality which ensures -linear convergence rates of the proposed method, incorporating various well-established conditions from the existing literature. Moreover we propose a -Bregman proximal point interpretation of the scheme that allows us to capture conditions that lead to sublinear rates under convexity.

Paper Structure

This paper contains 21 sections, 21 theorems, 109 equations, 2 figures, 1 table.

Table of Contents

  1. AMS Subject Classification:
  2. Introduction
  3. Motivation
  4. Paper organization
  5. Notation and preliminaries
  6. Φ-convexity
  7. Difference of Φ-convex problems
  8. Problem definition
  9. Difference of Φ-convex duality
  10. Optimality conditions for difference of Φ-convex problems
  11. The difference of Φ-convex algorithm
  12. Examples of Φ-convexity and unification of existing algorithms
  13. Convergence analysis
  14. Asymptotic convergence analysis
  15. Sublinear convergence rate analysis
  16. ...and 6 more sections

Key Result

Proposition 2.4

Let $X$ and $Y$ be nonempty sets, $\Phi: X \times Y \to \mathbb{R}$ a real-valued coupling and $f:X\to \overline{\mathbb{R}}$. Then we have In addition, $f^{\Phi\Phi}$ is the pointwise largest $\Phi$-convex function below $f$. In particular, this means that $f$ is $\Phi$-convex on $X$ if and only if $f(x) = f^{\Phi\Phi}(x)$ for all $x \in X$. The statements for $g:Y \to \overline{\mathbb{R}}$ are

Figures (2)

  • Figure 1: Illustration of the nonconvex function $f(x) = \max\{x^2 - x + 1, -x^2-5x+7\}$
  • Figure 2: Illustration of the $\Phi$-DC decomposition of the function $F(x) := g(x)-f(x)$ with $g(x):=\tfrac{1}{2}|x-\tfrac{1}{2}| + \delta_{[-4,4]}(x)$, $f(x):=-|x|^{1.5} \text{ if } x \in (-1, 1), \tfrac{1}{2}x^2-\tfrac{5}{2}|x|+1 \text{ otherwise}$ and $\Phi(x,y)=-\tfrac{3}{2}|\tfrac{x-y}{2}|^{1.5}$ and application of the proposed scheme: $\bar{x}$ is a point in $X$ and $x^+$ is the corresponding $\Phi$-DCA step. Starting from the top left corner and continuing clockwise: the function $F$; $g$ along with its $\Phi$-minorant; function $f$ along with its $\Phi$-minorant; the sum of both gap functions as the decrease of $F$.

Theorems & Definitions (65)

  • Definition 2.1: $\Phi$-convex and $\Phi$-concave functions
  • Definition 2.2: $\Phi$-conjugate functions
  • Definition 2.3: $\varepsilon$-$\Phi$-subgradients
  • Proposition 2.4
  • Proposition 2.5
  • Definition 2.6: selection
  • Example 2.7: quadratic coupling
  • Proposition 3.1
  • proof
  • Proposition 3.2: necessary and sufficient condition for global optimality
  • ...and 55 more