The Ingredients of Real-World Robotic Reinforcement Learning

TL;DR

The paper tackles the challenge of real-world robotic reinforcement learning without instrumentation or manual resets. It introduces R3L, a practical system that learns from raw sensory input, derives rewards via a goal-image discriminator (VICE), and trains in a reset-free, non-episodic setting using a randomized perturbation controller and unsupervised representation learning. Through simulation and real-world experiments on a three-fingered D'Claw, the authors show that this combination enables autonomous, vision-based manipulation skills with minimal human intervention, outperforming ablations and baseline approaches. The work advances scalable, autonomous embodied learning and points to future directions in safety, efficiency, and continual adaptation in open-world robotics.

Abstract

The success of reinforcement learning for real world robotics has been, in many cases limited to instrumented laboratory scenarios, often requiring arduous human effort and oversight to enable continuous learning. In this work, we discuss the elements that are needed for a robotic learning system that can continually and autonomously improve with data collected in the real world. We propose a particular instantiation of such a system, using dexterous manipulation as our case study. Subsequently, we investigate a number of challenges that come up when learning without instrumentation. In such settings, learning must be feasible without manually designed resets, using only on-board perception, and without hand-engineered reward functions. We propose simple and scalable solutions to these challenges, and then demonstrate the efficacy of our proposed system on a set of dexterous robotic manipulation tasks, providing an in-depth analysis of the challenges associated with this learning paradigm. We demonstrate that our complete system can learn without any human intervention, acquiring a variety of vision-based skills with a real-world three-fingered hand. Results and videos can be found at https://sites.google.com/view/realworld-rl/

Figures (16)

• Figure 1: Illustration of our proposed instrumentation-free system requiring minimal human engineering. Human intervention is only required in the goal collection phase (1). The robot is left to train unattended (2) during the learning phase and can be evaluated from arbitrary initial states at the end of training (3). We show sample goal and intermediate images from the training process of a real hardware system
• Figure 2: We draw a comparison between current real world learning systems which rely on instrumentation versus a system that learns in an environment more representative of the real world, free of instrumentation. While all three prior works utilize instrumentation for resets, state estimation, and reward, the motion capture system of Gupta2016LearningDM, sensor attached to the door in zhu2019dexterous, and auxiliary robot which picks up fallen balls in pddm are good examples of engineered state estimation, reward estimation, and reset mechanisms respectively.
• Figure 3: Our object repositioning task. The goal is to move the object from any starting configuration to a particular goal position and orientation.
• Figure 4: We report the approximate number of samples needed for a policy learned with a prior off-policy RL algorithm (SAC) to achieve average training performance of less than $0.15$ in pose distance (defined in Appendix \ref{['appendix:free_obj_repo']}) across 3 seeds on the re-positioning task. We compare training performance after varying three axes: ground truth rewards vs. learned rewards, with vs. without episodic resets, low-level state vs. images as inputs. We observe learning without resets is harder than with resets and is much harder when combined with visual inputs.
• Figure 5: We observe that when training reset free to reach a single goal, while the pose distance at training time is quite low, the pose errors obtained at test-time with the learned policy are very high. This indicates that while the object is getting close to the goal at training time, the policies being learned are still not effective.