Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL

TL;DR

MOLe addresses continual online adaptation under non-stationary task distributions by learning a mixture of meta-trained networks and updating them online via EM with a Chinese restaurant process prior. The meta-trained initialization enables rapid gradient-based adaptation, while the EM procedure allocates or recalls task components as the data stream evolves, supporting both generalization and specialization. Applied to model-based RL, MOLe demonstrates superior online adaptation to terrain changes, motor malfunctions, and leg impairment compared to k-shot and naive online gradient baselines, with the ability to recall past tasks. This approach offers a practical framework for robust, continual learning in robotics and other streaming domains without requiring predefined task boundaries.

Abstract

Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.

Figures (10)

• Figure 1: Half-cheetah agent, shown traversing a landscape with 'basins'
• Figure 2: Results on half-cheetah terrain traversal. The poorly performing model-based RL shows that a single model is not sufficient, and model-based RL with online gradient updates shows that a meta-learned initialization is critical. The three meta-learning approaches perform similarly; however, the performance of k-shot adaptation deteriorates when the task calls for taking multiple gradient steps away from the prior.
• Figure 3: Latent task distribution over time for two half-cheetah landscape traversal tasks, where encountered terrain slopes vary within each run. Interestingly, we find that MOLe chooses to only use a single latent task variable to describe varying terrain.
• Figure 4: Results on the motor malfunction trials, where different trials are shown task distributions that modulate at different frequencies (or stay constant, for the first category). Here, online learning is critical for good performance, k-shot adaptation is insufficient for such different tasks, and continued gradient steps leads to overfitting to recently seen data.
• Figure 5: Latent task variable distribution over the course of an online learning trial where the underlying motor malfunction changes every $500$ timesteps. We find that MOLe is able to successfully recover the task structure, recognize when the underlying task has changed, and recall previously seen tasks.