Are Greedy Task Orderings Better Than Random in Continual Linear Regression?

TL;DR

This work examines how the sequence of tasks affects learning in continual linear regression under joint realizability, introducing two greedy orderings—Maximum Distance and Maximum Residual—built on a projection/Kaczmarz view. It provides both empirical evidence and theoretical guarantees: greedy orderings can converge faster than random data, and in rank-$d-1$ settings they achieve a distance bound and a loss bound of $\mathcal{O}\left(\tfrac{1}{eT}\right)$, yet single-pass greedy can catastrophically fail under general rank. The authors address these failures by showing that repetitions restore a dimensionality-independent rate of $\mathcal{O}\left(k^{-1/3}\right)$ and by proposing a hybrid approach that blends greedy and random strategies, preserving favorable bounds. Across synthetic regression and CIFAR-100 linear probing, dissimilar-adjacent task sequences consistently outperform random orderings, illustrating practical benefits for curriculum-like task sequencing in continual learning. The work also delineates regimes where greedy strategies may fail and offers principled remedies, contributing a rigorous, Kaczmarz-informed framework for understanding and optimizing task orderings in continual learning.

Abstract

We analyze task orderings in continual learning for linear regression, assuming joint realizability of training data. We focus on orderings that greedily maximize dissimilarity between consecutive tasks, a concept briefly explored in prior work but still surrounded by open questions. Using tools from the Kaczmarz method literature, we formalize such orderings and develop geometric and algebraic intuitions around them. Empirically, we demonstrate that greedy orderings converge faster than random ones in terms of the average loss across tasks, both for linear regression with random data and for linear probing on CIFAR-100 classification tasks. Analytically, in a high-rank regression setting, we prove a loss bound for greedy orderings analogous to that of random ones. However, under general rank, we establish a repetition-dependent separation. Specifically, while prior work showed that for random orderings, with or without replacement, the average loss after $k$ iterations is bounded by $\mathcal{O}(1/\sqrt{k})$, we prove that single-pass greedy orderings may fail catastrophically, whereas those allowing repetition converge at rate $\mathcal{O}(1/\sqrt[3]{k})$. Overall, we reveal nuances within and between greedy and random orderings.

Figures (26)

• Figure 1: Intuition. Consider a collection of jointly-realizable linear regression tasks (e.g., A,B,C,D). Each task has an affine solution space (e.g., where $\mathbf{X}_{\texttt{A}}\mathbf{w} = \mathbf{y}_{\texttt{A}}$), and $\mathbf{w}_{\star}$ is an "offline" joint solution at the intersection of all tasks. Employing a projection perspective on learning in continual models evron2022catastrophicevron23continualClassification, we see that transitions between dissimilar tasks (A$\to$D$\to$B$\to$C) intuitively lead to faster convergence toward the intersection compared to transitions between similar tasks (A$\to$B$\to$C$\to$D).
• Figure 2: Projection dynamics.
• Figure 3: Task ordering comparison. Transitioning between dissimilar tasks consistently outperforms random transitions, with Greedy MR and MD achieving comparable performance.
• Figure 4: Learning an adversarial collection.
• Figure 5: The effect of repetitions.

Theorems & Definitions (128)

• Definition 2.2: Minimum-norm joint solution
• Definition 2.3: Average loss
• Remark 2.4: Forgetting vs. loss
• Definition 2.5: Distance to the joint solution
• Proposition 2.6: Linking the quantities
• Definition 3.1: Maximum Distance Ordering
• Definition 3.2: Maximum Residual Ordering
• Lemma 4.1: Optimality guarantee when $r=d-1$
• Lemma 4.2: Loss bound when $r=d-1$
• Example 5.1: Adversarial 3d construction