Non-Uniform Memory Sampling in Experience Replay

TL;DR

The paper investigates whether the standard assumption of uniform replay sampling in continual learning holds, and finds non-uniform replay can outperform uniform under identical buffer content. It conducts a controlled experimental study across CIFAR-10 and Imagenette with fixed reservoir-sampled buffers and 50 random non-uniform weightings, comparing to a uniform baseline. The best non-uniform distributions deliver consistent accuracy gains (e.g., a few percentage points) across buffer sizes and datasets, indicating potential gains from adaptive replay policies. These results motivate future work on online estimation of sample utility, adaptive weighting, and integrating replay weighting with buffer-management strategies to better mitigate catastrophic forgetting.

Abstract

Continual learning is the process of training machine learning models on a sequence of tasks where data distributions change over time. A well-known obstacle in this setting is catastrophic forgetting, a phenomenon in which a model drastically loses performance on previously learned tasks when learning new ones. A popular strategy to alleviate this problem is experience replay, in which a subset of old samples is stored in a memory buffer and replayed with new data. Despite continual learning advances focusing on which examples to store and how to incorporate them into the training loss, most approaches assume that sampling from this buffer is uniform by default. We challenge the assumption that uniform sampling is necessarily optimal. We conduct an experiment in which the memory buffer updates the same way in every trial, but the replay probability of each stored sample changes between trials based on different random weight distributions. Specifically, we generate 50 different non-uniform sampling probability weights for each trial and compare their final accuracy to the uniform sampling baseline. We find that there is always at least one distribution that significantly outperforms the baseline across multiple buffer sizes, models, and datasets. These results suggest that more principled adaptive replay policies could yield further gains. We discuss how exploiting this insight could inspire new research on non-uniform memory sampling in continual learning to better mitigate catastrophic forgetting. The code supporting this study is available at $\href{https://github.com/DentonJC/memory-sampling}{https://github.com/DentonJC/memory-sampling}$.