Jump-Start Reinforcement Learning

TL;DR

JSRL introduces a two-policy curriculum to bootstrap value-based RL using a pre-existing guide-policy, enabling efficient exploration by progressively rolling in with the guide-policy and transitioning to an exploration-policy. The approach yields polynomial sample complexity under a weak coverage assumption and demonstrates strong data-efficiency gains on simulated robotic tasks and vision-based manipulation, often outperforming IL+RL baselines in low-data regimes. The work provides both theoretical bounds and extensive empirical validation, highlighting practical impact for leveraging offline data or demonstrations in RL. These results suggest a robust, generic method for accelerating RL with prior policies, with potential real-world deployments and extensions to various guide-policy sources.

Abstract

Reinforcement learning (RL) provides a theoretical framework for continuously improving an agent's behavior via trial and error. However, efficiently learning policies from scratch can be very difficult, particularly for tasks with exploration challenges. In such settings, it might be desirable to initialize RL with an existing policy, offline data, or demonstrations. However, naively performing such initialization in RL often works poorly, especially for value-based methods. In this paper, we present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy, and is compatible with any RL approach. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks: a guide-policy, and an exploration-policy. By using the guide-policy to form a curriculum of starting states for the exploration-policy, we are able to efficiently improve performance on a set of simulated robotic tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms, particularly in the small-data regime. In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.

Figures (15)

• Figure 1: We study how to efficiently bootstrap value-based RL algorithms given access to a prior policy. In vanilla RL (left), the agent explores randomly from the initial state until it encounters a reward (gold star). JSRL (right), leverages a guide-policy (dashed blue line) that takes the agent closer to the reward. After the guide-policy finishes, the exploration-policy (solid orange line) continues acting in the environment. As the exploration-policy improves, the influence of the guide-policy diminishes, resulting in a learning curriculum for bootstrapping RL.
• Figure 2: Naïve policy initialization. We pre-train a policy to medium performance (depicted by negative steps), then use this policy to initialize actor-critic fine-tuning (starting from step 0), while initializing the critic randomly. Actor performance decays, as the untrained critic provides a poor learning signal, causing the good initial policy to be forgotten. In Figures \ref{['fig:additional_naive_bootstrap_100k_samples']} and \ref{['fig:additional_naive_bootstrap_1m_samples']}, we repeat this experiment but allow the randomly initialized critic to "warm up" before fine-tuning.
• Figure 3: We evaluate the importance of guide-policy quality for JSRL on Instance Grasping, the most challenging task we consider. By limiting the initial demonstrations, JSRL is less sensitive to limitations of initial demonstrations compared to baselines, especially in the small-data regime. For each of these initial demonstration settings, we find that QT-Opt+JSRL is more sample efficient than QT-Opt+JSRL-Random in early stages of training, but converge to the same final performances. A similar analysis for Indiscriminate Grasping is provided in Fig. \ref{['fig:additional_indisc_grasping_random']} in the Appendix.
• Figure 4: IL+RL methods on two simulated robotic grasping tasks. The baselines show improvement with fine-tuning, but QT-Opt+JSRL is more sample efficient and attains higher final performance. Each line depicts the mean and standard deviation over three random seeds.
• Figure 5: In the simulated vision-based robotic grasping tasks, a robot arm must grasp various objects placed in bins in front of it. Full implementation details are described in Appendix \ref{['appendix:manipulationenvironment']}.

Theorems & Definitions (7)

• Theorem 4.1: koenig1993complexity
• Theorem 4.3: Informal
• Theorem 1.3
• Corollary 1.4
• Corollary 1.5
• Remark 1.7
• proof