MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning

TL;DR

This paper tackles offline pre-training followed by online fine-tuning for reinforcement learning in high-dimensional visual robot tasks. It introduces MOTO, a model-based actor-critic algorithm that uses variational latent dynamics, model-based value expansion with horizon $H$, ensemble-based epistemic uncertainty with penalty $\alpha$, and behaviour-prior policy regularization to safely leverage offline data during online adaptation. Empirically, MOTO achieves state-of-the-art performance on 9 of 10 MetaWorld tasks and solves Franka Kitchen tasks from vision alone, illustrating strong generalization and combinatorial problem-solving from pixel inputs. The work provides theoretical and empirical validation of offline model-based performance bounds and positions MOTO as a scalable backbone for future model-based imitation and transfer learning in robotics.

Abstract

We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks. Recent offline model-free approaches successfully use online fine-tuning to either improve the performance of the agent over the data collection policy or adapt to novel tasks. At the same time, model-based RL algorithms have achieved significant progress in sample efficiency and the complexity of the tasks they can solve, yet remain under-utilized in the fine-tuning setting. In this work, we argue that existing model-based offline RL methods are not suitable for offline-to-online fine-tuning in high-dimensional domains due to issues with distribution shifts, off-dynamics data, and non-stationary rewards. We propose an on-policy model-based method that can efficiently reuse prior data through model-based value expansion and policy regularization, while preventing model exploitation by controlling epistemic uncertainty. We find that our approach successfully solves tasks from the MetaWorld benchmark, as well as the Franka Kitchen robot manipulation environment completely from images. To the best of our knowledge, MOTO is the first method to solve this environment from pixels.

Figures (7)

• Figure 1: Model-based offline to online fine-tuning. A static dataset of experience is used to train a world model, with which the offline actor-critic agent interacts. The actor-critic agent is trained via both data from the environment and data from the model via model-based value expansion. Model data is penalized via an uncertainty penalty which inhibits model exploitation. Finally, during fine-tuning, the agent interacts with the environment, and collects new trajectories, which are used to jointly fine-tune the world model and actor-critic.
• Figure 2: The success rates across the 10 MetaWorld tasks. MOTO matches or outperforms other methods on 9 out of the 10 tasks, demonstrating MOTO's ability to successfully pre-train offline and fine-tuning online on a variety of manipulation tasks using limited offline data. DreamerV2 is the only other method to achieve competitive results on the MetaWorld tasks. The model free baselines achieve low to moderate performance across all tasks.
• Figure 3: (Left) Success rate of completing the "mixed" and "partial" tasks in Franka Kitchen. MOTO outperforms all methods on both tasks, and is the only method to achieve meaningful progress on the "partial" task, indicating MOTO's capacity for combinatorial generalization. (Right) We carry out ablations on the MOTO design: no uncertainty penalties "No Unc.", no behavioral cloning regularization "No. BC", and removing both "No BC., No Unc."; removing model-based value expansion as well gives us DreamerV2. We observe that the gains from each component are additive, and only the full model achieves the best performance. Lastly, since all ablations share the same architecture, this shows that the performance improvement is not due to a stronger architecture, but rather the actor critic training.
• Figure 4: We evaluate the model's generalization capabilities at the end of the offline pre-training phase. The model correctly predicts rewards of up to 4 on successful episodes in the "partial" task, even though the maximum dataset reward is 3. (left). When doing rollouts in the learned model, the policy solves all four objects in the "partial" task and reaches rewards of up to 4 (right).
• Figure 5: Training curves for data ablation experiments. We see no degradation in performance when using only 100 and 250 pre-training episodes.

Theorems & Definitions (2)

• Theorem B.1
• proof