What Matters for Batch Online Reinforcement Learning in Robotics?

TL;DR

This paper addresses how to achieve effective batch online reinforcement learning in robotics by learning from large batches of autonomously collected data. It systematically develops and tests a recipe that combines an expressive imitation-learning policy as the actor, a value-based Q-function learned from autonomous data, and implicit policy extraction to guide rollouts, with an optional Ornstein-Uhlenbeck noise added to boost data diversity. Across six simulated robotic manipulation tasks and a real-world tape-hanging task, the approach yields up to twofold performance gains and better data efficiency than prior imitation-based methods. The findings underscore the importance of policy expressivity and implicit extraction in leveraging diverse autonomous data for scalable robot learning, offering a practical roadmap for practitioners and a basis for future research.

Abstract

The ability to learn from large batches of autonomously collected data for policy improvement -- a paradigm we refer to as batch online reinforcement learning -- holds the promise of enabling truly scalable robot learning by significantly reducing the need for human effort of data collection while getting benefits from self-improvement. Yet, despite the promise of this paradigm, it remains challenging to achieve due to algorithms not being able to learn effectively from the autonomous data. For example, prior works have applied imitation learning and filtered imitation learning methods to the batch online RL problem, but these algorithms often fail to efficiently improve from the autonomously collected data or converge quickly to a suboptimal point. This raises the question of what matters for effective batch online RL in robotics. Motivated by this question, we perform a systematic empirical study of three axes -- (i) algorithm class, (ii) policy extraction methods, and (iii) policy expressivity -- and analyze how these axes affect performance and scaling with the amount of autonomous data. Through our analysis, we make several observations. First, we observe that the use of Q-functions to guide batch online RL significantly improves performance over imitation-based methods. Building on this, we show that an implicit method of policy extraction -- via choosing the best action in the distribution of the policy -- is necessary over traditional policy extraction methods from offline RL. Next, we show that an expressive policy class is preferred over less expressive policy classes. Based on this analysis, we propose a general recipe for effective batch online RL. We then show a simple addition to the recipe of using temporally-correlated noise to obtain more diversity results in further performance gains. Our recipe obtains significantly better performance and scaling compared to prior methods.

Figures (15)

• Figure 1: Overview. We consider the batch online RL problem setting, in which a policy is trained on an initial dataset, used to collect batches of autonomous data during deployment, and then re-trained on the accumulated dataset. We analyze three critical axes in a spectrum of approaches to the batch online RL problem: policy expressivity, algorithm class, and policy extraction method. We propose a general effective recipe of training an expressive IL policy as the actor, value-based RL to learn a Q-function, and performing implicit policy extraction with the Q-function to get a policy for rollouts.
• Figure 2: Simulation environments. Robomimic tasks: Lift, Can, Square; MimicGen tasks: Threading, Stack; Adroit tasks: Pen.
• Figure 3: Normalized returns of different algorithm classes over multiple iterations of improvement. Value-based RL significantly outperforms IL and filtered-IL. Runs are 3 seeds, 100 evaluations.
• Figure 4: Heatmap of the state visitations of successful trajectories after batch online RL for value-based RL and filtered-IL on Lift and Square. A 3D plot of end-effector positions as well as a 2D cross-section are shown for each task. Darker colors correspond to higher density of visitation. Note that value-based RL methods achieve more state diversity in successful rollouts.
• Figure 5: Normalized returns of different algorithm classes at various data scales averaged across all tasks.