Statistical Context Detection for Deep Lifelong Reinforcement Learning

TL;DR

This work tackles online context detection in lifelong reinforcement learning where explicit task labels are unavailable. It introduces SWOKS, a method that combines Sliced Wasserstein Distance-based task distance with an online Kolmogorov-Smirnov test to detect and re-detect task changes, coupled with a rollback mechanism and modulating masks to maintain task-specific policies. The approach is validated on benchmarks including CT-graph, MiniGrid, and Half-Cheetah, showing robustness to changes in rewards, transitions, and even observation spaces, and offering a statistically principled means to allocate data to appropriate policies. The results indicate that optimal transport statistics provide explainable, justifiable cues for online context detection and reward optimization in lifelong reinforcement learning, with tunable parameters to balance false positives and detection speed across domains.

Abstract

Context detection involves labeling segments of an online stream of data as belonging to different tasks. Task labels are used in lifelong learning algorithms to perform consolidation or other procedures that prevent catastrophic forgetting. Inferring task labels from online experiences remains a challenging problem. Most approaches assume finite and low-dimension observation spaces or a preliminary training phase during which task labels are learned. Moreover, changes in the transition or reward functions can be detected only in combination with a policy, and therefore are more difficult to detect than changes in the input distribution. This paper presents an approach to learning both policies and labels in an online deep reinforcement learning setting. The key idea is to use distance metrics, obtained via optimal transport methods, i.e., Wasserstein distance, on suitable latent action-reward spaces to measure distances between sets of data points from past and current streams. Such distances can then be used for statistical tests based on an adapted Kolmogorov-Smirnov calculation to assign labels to sequences of experiences. A rollback procedure is introduced to learn multiple policies by ensuring that only the appropriate data is used to train the corresponding policy. The combination of task detection and policy deployment allows for the optimization of lifelong reinforcement learning agents without an oracle that provides task labels. The approach is tested using two benchmarks and the results show promising performance when compared with related context detection algorithms. The results suggest that optimal transport statistical methods provide an explainable and justifiable procedure for online context detection and reward optimization in lifelong reinforcement learning.

Figures (7)

• Figure 1: Graphical representation of the SWOKS architecture.
• Figure 2: Reward achieved over sequences of tasks. a) Average reward over 5 seeds for SWOKS and TFCL in the CT-graph environment. The sequence of tasks from 1 to 4 is seen twice in the order 1-2-3-4-1-2-3-4. The rolling average over 10 iterations is plotted for easier viewing. b) SWOKS, MBCD and 3RL are tested on the Half-Cheetah environment. Task changes occur every 40000 timesteps (40 iterations).
• Figure 3: Detected task over time for 5 seeds of the SWOKS agent performing in the CT-graph environment. The ground truth (GT) task is compared with the task label generated by each agent. Task labels given in this graph are from the same run as Figure \ref{['fig:tfcl']}. The label "None" corresponds to the agent having a low $p$-value for all previously seen tasks.
• Figure 4: Analysis of learning dynamics on the Minigrid environment for two seed runs. Task changes occur every 1000 iterations, between 3 tasks (details in appendix). The three tasks are each seen twice in the order 1-2-3-1-2-3. (a) Average reward: the first two tasks are learned, but the third one fails. (b) The $p$-values are high (above $0.001 = 10^{-3}$) when a task is detected. We observe that task 2 is mistakenly identified as task 1 (top left graph) but without affecting the performance. Task 3 is also correctly identified despite the agent failing to learn it.
• Figure 5: Plot over time of $p$-values for task label 1 and for different values of $\beta$ in the CT-graph benchmark. The curriculum is formed of tasks 1-2-3-4-1-2-3-4. We can observe that task 1 is correctly identified for all settings, but with $\beta=1.4$, SWOKS also believes tasks 2, 3, and 4 to be tasks 1. A rolling mean is taken over a sliding window of size 20 for readability.