Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chunking: Continual Learning is not just about Distribution Shift

Thomas L. Lee, Amos Storkey

TL;DR

This paper isolates the chunking sub-problem of continual learning by removing task-shift and shows that learning from identically distributed chunks accounts for a large portion of the gap between offline training and CL. It demonstrates that current CL methods offer little advantage over plain SGD in the chunking setting and that forgetting is a central issue even without distribution shift. To address this, it introduces per-chunk weight averaging (mean/EMA), which consistently improves chunking performance across multiple datasets and transfers to full CL with task shift. The findings suggest that tackling chunking can yield broad improvements in CL practice, and that practical techniques like weight averaging, possibly combined with pretraining, are valuable levers for progress.

Abstract

Work on continual learning (CL) has thus far largely focused on the problems arising from shifts in the data distribution. However, CL can be decomposed into two sub-problems: (a) shifts in the data distribution, and (b) dealing with the fact that the data is split into chunks and so only a part of the data is available to be trained on at any point in time. In this work, we look at the latter sub-problem, the chunking of data. We show that chunking is an important part of CL, accounting for around half of the performance drop from offline learning in our experiments. Furthermore, our results reveal that current CL algorithms do not address the chunking sub-problem, only performing as well as plain SGD training when there is no shift in the data distribution. Therefore, we show that chunking is both an important and currently unaddressed sub-problem and until it is addressed CL methods will be capped in performance. Additionally, we analyse why performance drops when learning occurs on identically distributed chunks of data, and find that forgetting, which is often seen to be a problem due to distribution shift, still arises and is a significant problem. We also show that performance on the chunking sub-problem can be increased and that this performance transfers to the full CL setting, where there is distribution shift. Hence, we argue that work on chunking can help advance CL in general.

Chunking: Continual Learning is not just about Distribution Shift

TL;DR

This paper isolates the chunking sub-problem of continual learning by removing task-shift and shows that learning from identically distributed chunks accounts for a large portion of the gap between offline training and CL. It demonstrates that current CL methods offer little advantage over plain SGD in the chunking setting and that forgetting is a central issue even without distribution shift. To address this, it introduces per-chunk weight averaging (mean/EMA), which consistently improves chunking performance across multiple datasets and transfers to full CL with task shift. The findings suggest that tackling chunking can yield broad improvements in CL practice, and that practical techniques like weight averaging, possibly combined with pretraining, are valuable levers for progress.

Abstract

Work on continual learning (CL) has thus far largely focused on the problems arising from shifts in the data distribution. However, CL can be decomposed into two sub-problems: (a) shifts in the data distribution, and (b) dealing with the fact that the data is split into chunks and so only a part of the data is available to be trained on at any point in time. In this work, we look at the latter sub-problem, the chunking of data. We show that chunking is an important part of CL, accounting for around half of the performance drop from offline learning in our experiments. Furthermore, our results reveal that current CL algorithms do not address the chunking sub-problem, only performing as well as plain SGD training when there is no shift in the data distribution. Therefore, we show that chunking is both an important and currently unaddressed sub-problem and until it is addressed CL methods will be capped in performance. Additionally, we analyse why performance drops when learning occurs on identically distributed chunks of data, and find that forgetting, which is often seen to be a problem due to distribution shift, still arises and is a significant problem. We also show that performance on the chunking sub-problem can be increased and that this performance transfers to the full CL setting, where there is distribution shift. Hence, we argue that work on chunking can help advance CL in general.
Paper Structure (21 sections, 2 theorems, 16 equations, 14 figures, 3 tables)

This paper contains 21 sections, 2 theorems, 16 equations, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Work
  3. The Chunking Setting
  4. Analysis of the Chunking Setting
  5. Performance in the Chunking setting
  6. Per-Chunk Weight Averaging
  7. Application to Continual Learning
  8. Conclusions
  9. Limitations
  10. Experimental Details
  11. Positive Transfer and the Chunking Problem
  12. The Chunking Problem in Online CL
  13. The Stability Gap in the Chunking Setting
  14. Analysis of the Linear Case
  15. The Effect of Pretraining in the Chunking Setting
  16. ...and 6 more sections

Key Result

Theorem 1

(proved in Appendix appen:proof) Assume that we have $k$ chunks and that each chunk $C_t = \{\mathbf{x}_i \in \mathbb{R}^{d} | i = 1,...,S\}$ is sampled i.i.d. from an $\alpha$-sub-Gaussian distribution (assuming zero mean) with a full rank covariance matrix $\mathbf{\Sigma}$. Also, assume bounded r with probability of at least $1- k a_2 e^{- a_3 S \min(\delta, \delta^2)}$. Defining $\lambda_d(\ma

Figures (14)

  • Figure 1: Standard continual learning versus the chunking setting. In standard continual learning (CL) a learner sequentially receives chunks of data called tasks and there is a shift in distribution between each task. While in the chunking setting, each chunk of data is identically distributed. CL methods have become better at dealing with task-shift and so partially address standard continual learning. However, we show that current CL methods do not tackle the chunking sub-problem as they perform no better than plain SGD training in the chunking setting. We also find that chunking contributes a significant part of the performance gap between offline learning and CL. This means that performance in the chunking setting is significantly lower than offline learning performance, for commonly used numbers of chunks. Therefore, as the chunking setting provides an approximate upperbound to performance in CL, improving capability in the chunking setting is a necessity to obtain high-performing CL methods.
  • Figure 2: End-of-training accuracy against chunk size on CIFAR-10 and CIFAR-100. Each data point on a curve presents the end-of-training accuracy of a method from a full run with chunks of the size given on the horizontal axis. The plots show that a smaller chunk size leads to a greater performance drop from offline learning (the performance of the right most point in each plot) and that CL methods perform similarly to plain SGD training in the chunking setting.
  • Figure 3: End-of-training accuracy against chunk size for Tiny ImageNet. Each data point on a curve presents the end-of-training accuracy of a method from a full run on Tiny ImageNet with chunks of the size given on the horizontal axis. The plot shows that the smaller the chunk size the greater the performance drop from offline learning (the performance of the right most point in the plot) and that CL methods perform similarly to plain SGD training in the chunking setting.
  • Figure 4: The training loss curve for plain SGD on CIFAR-100 when training on 50 chunks, where we plot the training loss for the first 2000 update steps corresponding to learning on the first 13 chunks. The plot shows that the loss converges for each chunk and hence that the learner does not underfit when training on any chunk.
  • Figure 5: Accuracy at the end of learning on each chunk for the training set of the $5^{th}$, $20^{th}$ and $40^{th}$ chunks and the test set, for CIFAR-100 and Tiny ImageNet when using plain SGD training. We split the datasets into 50 chunks, corresponding to a chunk size of 1000 and 2000 for CIFAR-100 and Tiny ImageNet, respectively. The plots show that after learning on a chunk the accuracy on that chunk quickly drops to the level of test set performance and hence that the learner quickly forgets a large part of the knowledge of a chunk after learning on it.
  • ...and 9 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2: rephrased from wainwright2019high (Theorem 6.5 p.166)