IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning

TL;DR

IN-RIL addresses instability and sample inefficiency in robotics policy fine-tuning by interleaving imitation learning updates with reinforcement learning updates, using gradient separation to prevent interference between $ \nabla_\theta \mathcal{L}_{IL}$ and $ \nabla_\theta \mathcal{L}_{RL}$. It establishes a theoretical characterization of the optimal interleaving ratio $m(t)$ and a positive regularization gain $\Delta_{IL-RL}$ that enable improved convergence and sample efficiency, then demonstrates substantial empirical gains across 14 tasks on FurnitureBench, Robomimic, and OpenAI Gym, including boosting Robomimic Transport success from 12% to 88% when paired with IDQL. The approach is algorithm-agnostic and acts as a plug-in for both on-policy (DPPO, residual PPO) and off-policy (IDQL) RL methods, with gradient separation delivered via gradient surgery or a residual-policy architecture. Overall, IN-RIL achieves greater stability, faster convergence, and better data efficiency, particularly in long-horizon, sparse-reward robotics tasks, highlighting the value of periodic IL guidance during fine-tuning for robust robotic control.

Abstract

Imitation learning (IL) and reinforcement learning (RL) each offer distinct advantages for robotics policy learning: IL provides stable learning from demonstrations, and RL promotes generalization through exploration. While existing robot learning approaches using IL-based pre-training followed by RL-based fine-tuning are promising, this two-step learning paradigm often suffers from instability and poor sample efficiency during the RL fine-tuning phase. In this work, we introduce IN-RIL, INterleaved Reinforcement learning and Imitation Learning, for policy fine-tuning, which periodically injects IL updates after multiple RL updates and hence can benefit from the stability of IL and the guidance of expert data for more efficient exploration throughout the entire fine-tuning process. Since IL and RL involve different optimization objectives, we develop gradient separation mechanisms to prevent destructive interference during \ABBR fine-tuning, by separating possibly conflicting gradient updates in orthogonal subspaces. Furthermore, we conduct rigorous analysis, and our findings shed light on why interleaving IL with RL stabilizes learning and improves sample-efficiency. Extensive experiments on 14 robot manipulation and locomotion tasks across 3 benchmarks, including FurnitureBench, OpenAI Gym, and Robomimic, demonstrate that \ABBR can significantly improve sample efficiency and mitigate performance collapse during online finetuning in both long- and short-horizon tasks with either sparse or dense rewards. IN-RIL, as a general plug-in compatible with various state-of-the-art RL algorithms, can significantly improve RL fine-tuning, e.g., from 12\% to 88\% with 6.3x improvement in the success rate on Robomimic Transport. Project page: https://github.com/ucd-dare/IN-RIL.

Figures (20)

• Figure 1: Comparison between IN--RIL (interleaved RL/IL) fine-tuning and RL fine-tuning on Transport, Round-Table, and Lamp, which are challenging multi-stage and sparse-reward tasks. Extensive experiments show that IL benefits from expert demonstrations but performance saturates at low success rates; and RL fine-tuning can suffer from stability and poor sample efficiency. IN--RIL fine-tuning succeeds to learn and outperforms RL fine-tuning by a significant margin in all tasks.
• Figure 2: An illustration of IN--RIL which updates the policy network with both IL and RL objectives.
• Figure 3: Optimization landscapes for IN--RIL. The IL loss landscape, represented by the 3D surface topology and its corresponding contour lines (where each contour connects points of equal IL loss value); and the landscape of RL rewards (or negative of the loss), represented by the color gradient mapped onto the surface (where the blue-to-white spectrum indicates low-to-high reward values as shown in the legend). IL updates drive the policy toward regions with lower losses, while RL updates steer toward higher rewards. Both optimization processes are stochastic and non-convex with multiple local optima. When using either RL or IL alone, training often converges to suboptimal solutions (as shown in the "IL only" and "RL only" trajectories). In contrast, our IN-RIL approach enables each objective to help escape the other's local optima: periodic IL updates help RL escape lower-reward regions toward higher-reward neighborhoods, while RL updates help IL traverse between different local minima in the loss landscape.
• Figure 4: An illustration of the two gradient separation mechanisms: a) gradient surgery, and b) network separation.
• Figure 5: Comparing IN--RIL with RL fine-tuning on Robomimic and Gym using DPPO.

Theorems & Definitions (11)

• Theorem 1: Optimal Interleaving Ratio
• Theorem 2: Iteration Complexity of IN--RIL
• Theorem 3: Convergence of RL-Only Training
• Theorem 4: Convergence with IN--RIL
• Lemma 1: Descent Lemma
• Lemma 2: Progress Bound for Gradient Descent
• Lemma 3: Error Bound for Stochastic Gradient Descent
• proof
• proof
• proof