FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning

TL;DR

FLaRe presents a scalable reinforcement learning fine-tuning framework that starts from a pre-trained multi-task robotics foundation model and tunes it via on-policy RL in large-scale simulations to overcome generalization limits of behavior cloning. By combining extensive simulated data, sparse task rewards, domain randomization, a frozen visual backbone (DinoV2), and stability techniques (small learning rate, entropy disablement, separate actor/critic, and KV-cache), FLaRe achieves state-of-the-art results on both seen and novel tasks and demonstrates robust sim-to-real transfer. The approach enables rapid adaptation to new embodiments and behaviors with modest fine-tuning and shows strong performance on real robots across a diverse set of tasks, highlighting a pathway toward generalizable and adaptable robotic systems. Limitations include reliance on simulators for fine-tuning and challenges when robust simulation is unavailable for certain domains, motivating future work on real-world RL fine-tuning and richer sim-to-real pipelines.

Abstract

In recent years, the Robotics field has initiated several efforts toward building generalist robot policies through large-scale multi-task Behavior Cloning. However, direct deployments of these policies have led to unsatisfactory performance, where the policy struggles with unseen states and tasks. How can we break through the performance plateau of these models and elevate their capabilities to new heights? In this paper, we propose FLaRe, a large-scale Reinforcement Learning fine-tuning framework that integrates robust pre-trained representations, large-scale training, and gradient stabilization techniques. Our method aligns pre-trained policies towards task completion, achieving state-of-the-art (SoTA) performance both on previously demonstrated and on entirely novel tasks and embodiments. Specifically, on a set of long-horizon mobile manipulation tasks, FLaRe achieves an average success rate of 79.5% in unseen environments, with absolute improvements of +23.6% in simulation and +30.7% on real robots over prior SoTA methods. By utilizing only sparse rewards, our approach can enable generalizing to new capabilities beyond the pretraining data with minimal human effort. Moreover, we demonstrate rapid adaptation to new embodiments and behaviors with less than a day of fine-tuning. Videos can be found on the project website at https://robot-flare.github.io/

Figures (7)

• Figure 1: FLaRe is a simple but effective approach for large-scale fine-tuning of robotic policies. FLaRe achieves SoTA performance on simulation (+23.6%) and real-world (+30.7%) benchmarks, can generalize to unseen tasks, and adapts to new behaviors and embodiments.
• Figure 2: FLaRe introduces a series of design choices that help stabilize the RL training process, including 1) fine-tuning from a multi-task robotics policy, 2) large-scale fine-tuning in simulation, 3) using an on-policy algorithm as opposed to off-policy methods, 4) utilizing smaller learning rate than when performing RL from scratch, 5) disabling the entropy bonus objective that can potentially distort the policy at the start of the training, and 6) separating the actor and the critic network, so that the critic update will not influence the policy prediction.
• Figure 3: FLaRe can efficiently fine-tune large transformer policies through large-scale Reinforcement Learning, using a sparse reward function that requires minimal human effort.
• Figure 4: We evaluate FLaRe on mobile manipulation tasks. (a, b) In-distribution tasks, in unseen environments. (c, d) Novel tasks that require unseen capabilities from pretraining, in unseen environments. FLaRe excels in long-horizon tasks, showing strong object recognition, relational reasoning, and exploration abilities.
• Figure 5: The real-world layout that we tested upon