Efficient Continual Finite-Sum Minimization

TL;DR

Continual finite-sum minimization extends standard finite-sum optimization to prefix objectives $g_i(x)=\frac{1}{i}\sum_{j=1}^i f_j(x)$ and seeks an $\epsilon$-accurate sequence $\hat{x}_i$. The paper introduces CSVRG, a first-order variance-reduction method that uses a sparsified full-gradient direction and a 3-FO gradient estimator to achieve a total FO complexity of $\tilde{O}\left(\frac{n}{\epsilon^{1/3}}+\frac{\log n}{\sqrt{\epsilon}}\right)$ in the strongly convex setting. It provides tight lower bounds for the class of natural first-order methods, analyzes the estimator's unbiasedness and variance, and demonstrates practical gains via ridge regression and MNIST-based experiments. Collectively, these results offer scalable, provably efficient continual optimization methods suitable for streaming data and lifelong learning scenarios.

Abstract

Given a sequence of functions $f_1,\ldots,f_n$ with $f_i:\mathcal{D}\mapsto \mathbb{R}$, finite-sum minimization seeks a point ${x}^\star \in \mathcal{D}$ minimizing $\sum_{j=1}^n f_j(x)/n$. In this work, we propose a key twist into the finite-sum minimization, dubbed as continual finite-sum minimization, that asks for a sequence of points ${x}_1^\star,\ldots,{x}_n^\star \in \mathcal{D}$ such that each ${x}^\star_i \in \mathcal{D}$ minimizes the prefix-sum $\sum_{j=1}^if_j(x)/i$. Assuming that each prefix-sum is strongly convex, we develop a first-order continual stochastic variance reduction gradient method ($\mathrm{CSVRG}$) producing an $ε$-optimal sequence with $\mathcal{\tilde{O}}(n/ε^{1/3} + 1/\sqrtε)$ overall first-order oracles (FO). An FO corresponds to the computation of a single gradient $\nabla f_j(x)$ at a given $x \in \mathcal{D}$ for some $j \in [n]$. Our approach significantly improves upon the $\mathcal{O}(n/ε)$ FOs that $\mathrm{StochasticGradientDescent}$ requires and the $\mathcal{O}(n^2 \log (1/ε))$ FOs that state-of-the-art variance reduction methods such as $\mathrm{Katyusha}$ require. We also prove that there is no natural first-order method with $\mathcal{O}\left(n/ε^α\right)$ gradient complexity for $α< 1/4$, establishing that the first-order complexity of our method is nearly tight.

Theorems & Definitions (72)

• Remark 1
• Definition 1: Strong Convexity
• Theorem 1
• Theorem 2
• Theorem 3
• Remark 2
• Definition 2
• Example 1
• Definition 3
• Remark 3