A Practical Guide to Streaming Continual Learning

TL;DR

Streaming Continual Learning is claimed to be an emerging paradigm providing a unifying solution to real-world problems, which may require both SML and CL abilities, and to foster the design of hybrid approaches that can quickly adapt to new information without forgetting previous knowledge.

Abstract

Continual Learning (CL) and Streaming Machine Learning (SML) study the ability of agents to learn from a stream of non-stationary data. Despite sharing some similarities, they address different and complementary challenges. While SML focuses on rapid adaptation after changes (concept drifts), CL aims to retain past knowledge when learning new tasks. After a brief introduction to CL and SML, we discuss Streaming Continual Learning (SCL), an emerging paradigm providing a unifying solution to real-world problems, which may require both SML and CL abilities. We claim that SCL can i) connect the CL and SML communities, motivating their work towards the same goal, and ii) foster the design of hybrid approaches that can quickly adapt to new information (as in SML) without forgetting previous knowledge (as in CL). We conclude the paper with a motivating example and a set of experiments, highlighting the need for SCL by showing how CL and SML alone struggle in achieving rapid adaptation and knowledge retention.

Figures (8)

• Figure 1: Overview of the structure of this manuscript. We start with an introduction to non-stationary environments, followed by a short background on streaming and continual learning, before discussing the streaming continual learning paradigm together with an empirical example motivating why it is needed.
• Figure 2: Illustrations of various concept drift types. The red line represents the theoretical decision boundary of the problem that the model must fit. \ref{['fig:real_drift']} depicts a real drift where the decision boundary shifts. \ref{['fig:virtual_zoom_in']} represents a zoom-in virtual drift that concentrates on a subset of the initial feature space. \ref{['fig:virtual_expansionary']} shows a pure expansionary virtual drift introducing a new, previously unseen feature subspace. \ref{['fig:virtual_mixed_expansionary']} displays a drift involving both familiar and new subspaces without altering the original decision boundary. Finally, \ref{['fig:virtual_mixed_real']} illustrates a combination of virtual and real drift, resulting in changes to the decision boundary within the previously known subspace.
• Figure 3: An RL control loop: as a reaction to actions $a(t)$ taken by the agent, it receives observations $o(t)$ and rewards $r(t)$.
• Figure 4: Virtual and real drift in Continual Reinforcement Learning (CRL). In this example scenario, a robot is rewarded or punished when touching certain objects, depending on form, color, and displayed symbol. Left: an object that should be touched, giving +10 reward. Middle: virtual drift, introducing a different object that should not be touched (or else: +10 reward). Right: real drift, changing the reward previously obtained for touching the object to punishment.
• Figure 5: Summary of SML and CL, with some examples of popular state-of-the-art approaches and popular libraries. We refer to existing reviews for an in-depth treatment of either field.