Replay-free Online Continual Learning with Self-Supervised MultiPatches

TL;DR

This work tackles Online Continual Learning (OCL) under privacy and data-availability constraints by removing the need for replay. It introduces Continual MultiPatches (CMP), a plug-in that creates $N$ patches per input and optimizes a joint loss $L_{CMP} = \beta L_{TCR}([z_1, \dots, z_N]) + \alpha \sum_{i=1}^N L_{SSL}(z_i, z_{avg})$ with Total Coding Rate $L_{TCR}$ to prevent collapse, enabling replay-free learning. The approach is demonstrated as a drop-in for SimSiam and BYOL (SimSiam-CMP and BYOL-CMP), achieving competitive or superior performance to replay-based OCSSL on Split CIFAR-100 and Split ImageNet100 under tight budgets. This indicates CMP’s potential as a scalable, privacy-preserving alternative that can enhance online SSL methods without external memory, with broad practical implications for real-world, data-restricted continual learning.

Abstract

Online Continual Learning (OCL) methods train a model on a non-stationary data stream where only a few examples are available at a time, often leveraging replay strategies. However, usage of replay is sometimes forbidden, especially in applications with strict privacy regulations. Therefore, we propose Continual MultiPatches (CMP), an effective plug-in for existing OCL self-supervised learning strategies that avoids the use of replay samples. CMP generates multiple patches from a single example and projects them into a shared feature space, where patches coming from the same example are pushed together without collapsing into a single point. CMP surpasses replay and other SSL-based strategies on OCL streams, challenging the role of replay as a go-to solution for self-supervised OCL.

Figures (1)

• Figure 1: Comparison between CMP (left) and ER (right) in OCSSL. While ER requires an external memory buffer, CMP only requires the current example $x$.