Adaptive Open-Loop Step-Sizes for Accelerated Convergence Rates of the Frank-Wolfe Algorithm

TL;DR

This work tackles constrained convex optimization via the Frank-Wolfe algorithm by replacing fixed-parameter open-loop step-sizes with a log-adaptive rule η_t = $\frac{2+\log(t+1)}{t+2+\log(t+1)}$, proving it matches or surpasses all fixed-$\ell$ strategies up to polylog factors. By formulating FW with adaptive step-sizes in the general form η_t = $\frac{g(t)}{t+g(t)}$ and imposing simple Assumptions A1–A2 on g, the authors extend the affine-invariant convergence blueprint to obtain accelerated convergence under strong $(M,r)$-growth and robust bounds under weak growth. In the strong-growth regime, the method achieves ${\mathsf{primaldual}}_{t} = \tilde{\mathcal{O}}(t^{-1/(1-r)})$, and in the weak-growth regime, ${\mathsf{subopt}}_{t} = \tilde{\mathcal{O}}(t^{-1/(1-r)} + t^{-2})$, with the log-adaptive choice enabling arbitrarily fast sublinear rates when $r$ is small. The results are corroborated by numerical experiments on constrained regression and collaborative filtering, and the log-adaptive rule is implemented in the FrankWolfe.jl package, offering a practical, computation-friendly default for FW step-sizes.

Abstract

Recent work has shown that in certain settings, the Frank-Wolfe algorithm (FW) with open-loop step-sizes $η_t = \frac{\ell}{t+\ell}$ for a fixed parameter $\ell \in \mathbb{N},\, \ell \geq 2$, attains a convergence rate faster than the traditional $O(t^{-1})$ rate. In particular, when a strong growth property holds, the convergence rate attainable with open-loop step-sizes $η_t = \frac{\ell}{t+\ell}$ is $O(t^{-\ell})$. In this setting there is no single value of the parameter $\ell$ that prevails as superior. This paper shows that FW with log-adaptive open-loop step-sizes $η_t = \frac{2+\log(t+1)}{t+2+\log(t+1)}$ attains a convergence rate that is at least as fast as that attainable with fixed-parameter open-loop step-sizes $η_t = \frac{\ell}{t+\ell}$ for any value of $\ell \in \mathbb{N},\,\ell\geq 2$. To establish our main convergence results, we extend our previous affine-invariant accelerated convergence results for FW to more general open-loop step-sizes of the form $η_t = g(t)/(t+g(t))$, where $g:\mathbb{N}\to\mathbb{R}_{\geq 0}$ is any non-decreasing function such that the sequence of step-sizes $(η_t)$ is non-increasing. This covers in particular the fixed-parameter case by choosing $g(t) = \ell$ and the log-adaptive case by choosing $g(t) = 2+ \log(t+1)$. To facilitate adoption of log-adaptive open-loop step-sizes, we have incorporated this rule into the {\tt FrankWolfe.jl} software package.

Figures (3)

• Figure 1: Constrained regression over the $\ell_2$-ball. Convergence rate comparison of FW with different step-sizes for \ref{['eq.regression']} for different locations of the unconstrained optimizer in the relative interior, on the relative boundary, and in the relative exterior of the feasible region and for the three different optimality measures ${\mathsf{gap}}_t$, ${\mathsf{primaldual}}_t$, and ${\mathsf{subopt}}_t$ on the Boston-housing dataset. Axes are in log scale.
• Figure 2: Constrained regression over the $\ell_5$-ball. Convergence rate comparison of FW with different step-sizes for \ref{['eq.regression']} for different locations of the unconstrained optimizer in the relative interior, on the relative boundary, and in the relative exterior of the feasible region and for the three different optimality measures ${\mathsf{gap}}_t$, ${\mathsf{primaldual}}_t$, and ${\mathsf{subopt}}_t$ on the Boston-housing dataset. Axes are in log scale.
• Figure 3: Collaborative filtering over nuclear norm balls of radii $1000$ and $3000$. Convergence rate comparison of FW with different step-sizes for \ref{['eq.collaborative_filterting']} over nuclear norm balls of radii $1000$ and $3000$ for the three different optimality measures ${\mathsf{gap}}_t$, ${\mathsf{primaldual}}_t$, and ${\mathsf{subopt}}_t$ on the movielens dataset. Axes are in log scale.

Theorems & Definitions (7)

• Lemma 3.1: Strengthened cumulative-product bound
• proof
• Theorem 3.2: Strong $(M,r)$-growth
• proof
• Theorem 4.1: Weak $(M, r)$-growth
• proof
• Remark 5.1: Open question: growth settings of collaborative filtering