Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A nonsmooth Frank-Wolfe algorithm through a dual cutting-plane approach

Guilherme Mazanti, Thibault Moquet, Laurent Pfeiffer

TL;DR

The DLS algorithm generates a sequence of primal-dual candidates, and it is proved that the corresponding primal-dual gap converges with a rate of O(1/t) and is extended to address a class of nonsmooth costs, involving in particular support functions.

Abstract

An extension of the Frank-Wolfe Algorithm (FWA), also known as Conditional Gradient algorithm, is proposed. In its standard form, the FWA allows to solve constrained optimization problems involving $β$-smooth cost functions, calling at each iteration a Linear Minimization Oracle. More specifically, the oracle solves a problem obtained by linearization of the original cost function. The algorithm designed and investigated in this article, named Dualized Level-Set (DLS) algorithm, extends the FWA and allows to address a class of nonsmooth costs, involving in particular support functions. The key idea behind the construction of the DLS method is a general interpretation of the FWA as a cutting-plane algorithm, from the dual point of view. The DLS algorithm essentially results from a dualization of a specific cutting-plane algorithm, based on projections on some level sets. The DLS algorithm generates a sequence of primal-dual candidates, and we prove that the corresponding primal-dual gap converges with a rate of $O(1/\sqrt{t})$.

A nonsmooth Frank-Wolfe algorithm through a dual cutting-plane approach

TL;DR

The DLS algorithm generates a sequence of primal-dual candidates, and it is proved that the corresponding primal-dual gap converges with a rate of O(1/t) and is extended to address a class of nonsmooth costs, involving in particular support functions.

Abstract

An extension of the Frank-Wolfe Algorithm (FWA), also known as Conditional Gradient algorithm, is proposed. In its standard form, the FWA allows to solve constrained optimization problems involving -smooth cost functions, calling at each iteration a Linear Minimization Oracle. More specifically, the oracle solves a problem obtained by linearization of the original cost function. The algorithm designed and investigated in this article, named Dualized Level-Set (DLS) algorithm, extends the FWA and allows to address a class of nonsmooth costs, involving in particular support functions. The key idea behind the construction of the DLS method is a general interpretation of the FWA as a cutting-plane algorithm, from the dual point of view. The DLS algorithm essentially results from a dualization of a specific cutting-plane algorithm, based on projections on some level sets. The DLS algorithm generates a sequence of primal-dual candidates, and we prove that the corresponding primal-dual gap converges with a rate of .
Paper Structure (41 sections, 30 theorems, 143 equations, 6 figures, 6 algorithms)

This paper contains 41 sections, 30 theorems, 143 equations, 6 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and notations
  3. General notations
  4. Duality
  5. Bregman distances
  6. Coercive functions
  7. Perspective functions
  8. The Extended Level-Set method
  9. A dual point of view on the fully-corrective FWA
  10. Statement of the ELS method
  11. Convergence analysis
  12. The pruning step
  13. The Dualized Level-Set method
  14. Framework and mathematical assumptions
  15. Structural assumptions
  16. ...and 26 more sections

Key Result

Theorem 1

Assume that $f\colon\mathcal{X}\to\bar{\mathbb R}^+$ and $f\not\equiv+\infty$. Then $f\in\symbpar*{\Gamma_0}{}{\mathcal{X}}$iff$f^{**}=f$, and in that case $f^*\in\symbpar*{\Gamma_0}{}{\mathcal{X}^*}$.

Figures (6)

  • Figure 1: Primal-dual gap for 4 instances of \ref{['expb:proj']}
  • Figure 2: Evolution of the number of cuts for 2 instances of \ref{['expb:proj']}
  • Figure 3: Primal-dual gap
  • Figure 4: Number of cuts at each iteration
  • Figure : FWA for \ref{['pb:FW_primal']}
  • ...and 1 more figures

Theorems & Definitions (59)

  • Theorem 1: name = Fenchel--Moreau,label = th:FenchelMoreau
  • Lemma 2: label=lem:FenchelYoungEq
  • Remark 3: label=rem:subgrad_support
  • Corollary 4: label = coro:subgrad_oracle
  • Theorem 5: name= Fenchel--Rockafellar, label=th:FenchelRockafellar
  • Corollary 6: label=coro:fenchel_rockafellar
  • proof
  • Remark 7: label=rem:breg_sqdist
  • Lemma 8: label=lem:identity_Breg
  • Remark 9
  • ...and 49 more