Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lower Bounds for Linear Minimization Oracle Methods Optimizing over Strongly Convex Sets

Benjamin Grimmer, Ning Liu

TL;DR

This work considers the oracle complexity of constrained convex optimization given access to a Linear Minimization Oracle (LMO) for the constraint set and a gradient oracle for the $L$-smooth, strongly convex objective, and establishes this as the optimal complexity for deterministic LMO methods over strongly convex constraint sets.

Abstract

We consider the oracle complexity of constrained convex optimization given access to a Linear Minimization Oracle (LMO) for the constraint set and a gradient oracle for the $L$-smooth, strongly convex objective. This model includes Frank-Wolfe methods and their many variants. Over the problem class of strongly convex constraint sets $S$, our main result proves that no such deterministic method can guarantee a final objective gap less than $\varepsilon$ in fewer than $Ω(\sqrt{L\, \mathrm{diam}(S)^2/\varepsilon})$ iterations. Our lower bound matches, up to constants, the accelerated Frank-Wolfe theory of Garber and Hazan (2015). Together, these establish this as the optimal complexity for deterministic LMO methods over strongly convex constraint sets. Second, we consider optimization over $β$-smooth sets, finding that in the modestly smooth regime of $β=Ω(1/\sqrt{\varepsilon})$, no complexity improvement for span-based LMO methods is possible against either compact convex sets or strongly convex sets.

Lower Bounds for Linear Minimization Oracle Methods Optimizing over Strongly Convex Sets

TL;DR

This work considers the oracle complexity of constrained convex optimization given access to a Linear Minimization Oracle (LMO) for the constraint set and a gradient oracle for the -smooth, strongly convex objective, and establishes this as the optimal complexity for deterministic LMO methods over strongly convex constraint sets.

Abstract

We consider the oracle complexity of constrained convex optimization given access to a Linear Minimization Oracle (LMO) for the constraint set and a gradient oracle for the -smooth, strongly convex objective. This model includes Frank-Wolfe methods and their many variants. Over the problem class of strongly convex constraint sets , our main result proves that no such deterministic method can guarantee a final objective gap less than in fewer than iterations. Our lower bound matches, up to constants, the accelerated Frank-Wolfe theory of Garber and Hazan (2015). Together, these establish this as the optimal complexity for deterministic LMO methods over strongly convex constraint sets. Second, we consider optimization over -smooth sets, finding that in the modestly smooth regime of , no complexity improvement for span-based LMO methods is possible against either compact convex sets or strongly convex sets.
Paper Structure (19 sections, 11 theorems, 58 equations)

This paper contains 19 sections, 11 theorems, 58 equations.

Table of Contents

  1. Introduction
  2. The Families of Smooth and Strongly Convex Constraint Sets and Functions.
  3. The Family of First-Order Linear Minimization Oracle Methods (FO-LMO).
  4. Our Contributions.
  5. Outline.
  6. Related Work
  7. Lower Bounds for Strongly Convex Sets and Span Methods
  8. A Family of First-Order Linear Minimization Oracle Span Methods (LMO-span).
  9. Construction of a Hard Problem Instance
  10. A Zero-Chain Property
  11. Proof of Theorem \ref{['thm:SC']}
  12. A Resisting Oracle Extension to Deterministic FO-LMO Methods
  13. A Permuted Family of Hard Problem Instances
  14. A Generalized Zero-Chain Property
  15. Proof of Theorem \ref{['thm:main']}
  16. ...and 4 more sections

Key Result

Theorem 1.1

Consider any $T \ge 1$ and $L,\alpha>0$. Then there exist an $\alpha$-strongly convex set $S$ and an $L$-smooth, $L$-strongly convex function $f$ in dimension $d=2(T+1)$ such that every FO-LMO method applied to $(f, S)$ with $x_0 = 0$ has $x_T$ infeasible or Consequently, for any $\varepsilon>0$, there exist problem instances where $T\geq \sqrt{\frac{L\, \mathrm{diam}(S)^2}{528\varepsilon}} - 1$

Theorems & Definitions (18)

  • Theorem 1.1
  • Theorem 2.1
  • Lemma 2.1: Strong Convexity of $S$
  • proof
  • Lemma 2.2: Exact Diameter of $S$
  • proof
  • Lemma 2.3: Exact LMO
  • proof
  • Lemma 2.4: Zero-Chain Property
  • proof
  • ...and 8 more