Beyond Short Steps in Frank-Wolfe Algorithms

TL;DR

This paper addresses projection-free convex optimization by improving Frank-Wolfe methods through primal-dual analyses and novel strategies. It introduces an optimistic Frank-Wolfe algorithm and a generalized primal-dual short-step framework, providing computable stopping criteria via the primal-dual gap and convergence guarantees that extend to gradient descent. The key contributions include the optimistic FW with OFTRL/OMD updates, a flexible primal-dual short-step scheme with line-search options, and refined primal-dual convergence rates along with empirical demonstrations of practical advantages. The work advances both the theoretical understanding and the applicability of FW methods, enabling faster, stopping-criterion-driven optimization in large-scale, constraint-heavy scenarios and potentially benefiting broader gradient-descent-based methods. Overall, the proposed methods offer tighter dual bounds and improved adaptability to curvature while preserving projection-free updates.

Abstract

We introduce novel techniques to enhance Frank-Wolfe algorithms by leveraging function smoothness beyond traditional short steps. Our study focuses on Frank-Wolfe algorithms with step sizes that incorporate primal-dual guarantees, offering practical stopping criteria. We present a new Frank-Wolfe algorithm utilizing an optimistic framework and provide a primal-dual convergence proof. Additionally, we propose a generalized short-step strategy aimed at optimizing a computable primal-dual gap. Interestingly, this new generalized short-step strategy is also applicable to gradient descent algorithms beyond Frank-Wolfe methods. As a byproduct, our work revisits and refines primal-dual techniques for analyzing Frank-Wolfe algorithms, achieving tighter primal-dual convergence rates. Empirical results demonstrate that our optimistic algorithm outperforms existing methods, highlighting its practical advantages.

Figures (6)

• Figure 1: Comparison over the probability simplex of dimension $n=1000$ with objective $f(x) = \| x - x_0 \|_2^2$, where $x_0$ is a random point outside the probability simplex. We can see that the optimistic variant converges faster than the other variants both in iterations and time. Note that we cut datapoints with excessively large primal/dual values, which leads to apparent different starting points in the graphs.
• Figure 2: Comparison over $k$-sparse polytope of dimension $n=100$ and $k = 10$ with objective $f(x) = \| Ax - b \|_2^2$, where $A$ and $b$ are random. We can see that the optimistic variant also converges faster than the other variants both in iterations and time. Here we also cut datapoints with excessively large primal/dual values, which leads to apparent different starting points in the graphs.
• Figure 3: Comparison over the probability simplex of dimension $n=1000$ with objective $f(x) = \| x - x_0 \|_2^2$, where $x_0$ is a random point outside the probability simplex. We see that primal-dual short steps converge exactly as fast as the vanilla variant.
• Figure 4: Comparison over the probability simplex of dimension $n=1000$ with objective $f(x) = \| x - x_0 \|_2^2$, where $x_0$ is a random point outside the probability simplex. FW+LB denotes the standard FW variant with the heavy ball lower bound. We can see that FW with the heavy ball lower bound converges no faster than the normal FW in dual gap.
• Figure 5: Comparison over the probability simplex of dimension $n = 100$ with objective $f(x) = \| Ax - b \|_2^2$, where $A$ and $b$ are random. We can see that the optimistic variant converges faster than the other variants both in iterations and time.

Theorems & Definitions (30)

• Remark 2.1: Alternative step-size strategies
• Theorem 3.1
• Proposition 4.1
• Proposition 4.2
• Remark 5.1: Strength of lower bounds
• Remark 1.1: Alternative step-sizes in Generalized Frank-Wolfe
• Remark 1.2: Arbitrary fast rate for $\psi(x^\ast)$
• Theorem 2.1: Optimistic FTRL
• proof
• Corollary 2.2