RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation

TL;DR

This work tackles high-precision real-world robotic manipulation by introducing RM-RL, a framework that unifies online RL with offline supervised learning through a role-model labeling mechanism. The key idea is to select an approximately optimal role-model action $a^*$ within similar initial states and use it to label online experiences for offline reuse, reframing learning as supervised updates and improving data efficiency and stability. A hybrid online-offline training schedule and a dedicated cell-plate localization pipeline enable rapid convergence and robust real-world performance, achieving 53% translation and 20% rotation improvements, and reliably placing a cell plate onto a shelf where prior methods struggle. The results demonstrate the practical impact of data-efficient, stable RL in precise manipulation tasks and set the stage for extending RM-RL to more complex, long-horizon robotics applications.

Abstract

Precise robot manipulation is critical for fine-grained applications such as chemical and biological experiments, where even small errors (e.g., reagent spillage) can invalidate an entire task. Existing approaches often rely on pre-collected expert demonstrations and train policies via imitation learning (IL) or offline reinforcement learning (RL). However, obtaining high-quality demonstrations for precision tasks is difficult and time-consuming, while offline RL commonly suffers from distribution shifts and low data efficiency. We introduce a Role-Model Reinforcement Learning (RM-RL) framework that unifies online and offline training in real-world environments. The key idea is a role-model strategy that automatically generates labels for online training data using approximately optimal actions, eliminating the need for human demonstrations. RM-RL reformulates policy learning as supervised training, reducing instability from distribution mismatch and improving efficiency. A hybrid training scheme further leverages online role-model data for offline reuse, enhancing data efficiency through repeated sampling. Extensive experiments show that RM-RL converges faster and more stably than existing RL methods, yielding significant gains in real-world manipulation: 53% improvement in translation accuracy and 20% in rotation accuracy. Finally, we demonstrate the successful execution of a challenging task, precisely placing a cell plate onto a shelf, highlighting the framework's effectiveness where prior methods fail.

Figures (5)

• Figure 1: The Ufactory X-ARM 6 autonomously executes a pick-and-place task, transferring a cell plate to the designated shelf. The red curve illustrates the trajectory of the end-effector.
• Figure 2: The framework of the proposed Role-Model Reinforcement Learning (RM-RL). The framework mainly includes three parts: the Online Real-world RL training, the Role-Model Online Labeling, and the Data Replay. In the online real-world RL part, actions are sampled to update the policy network using a policy gradient under similar scenes and initial states. In the role-model online labeling part, the approximately optimal action $a_i^*$ is selected as a label for the rest of the states. The labeled data are collected as the sub-dataset $\mathcal{D}_i$ to build a dataset $\mathcal{D}$ for further training. In the data replay part, actions and states are sampled from the $\mathcal{D}$ to update the policy network.
• Figure 3: The Policy network. The network takes the global image and the estimated picking pose as inputs, and outputs probability distributions over $\Delta x$, $\Delta y$, and $\Delta \psi$. The final pose adjustments are sampled from these distributions to refine the picking pose.
• Figure 4: The real-world experimental setting. The setting includes robotic arm (X-ARM 6), cell plate, cell plate shelf, and overhead camera (Intel RealSense D435). The red stars represent the target positions for the precise picking and placing tasks.
• Figure 5: The results of reward curves. The reward curves record the reward changes during real-world RL training from the selected baselines, Standard RL, standard RL with replay buffer (RL + Replay Buffer), the proposed Role-Model Reinforcement Learning (RM-RL), and pretrained RM-RL.