Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies

TL;DR

This work proposes an effective strategy for object insertion under tight tolerances that integrates traditional model-based control with RL to achieve improved insertion accuracy and outperforms recent RL-based methods in this domain and prior efforts with hybrid policies.

Abstract

Object insertion under tight tolerances ($< \hspace{-.02in} 1mm$) is an important but challenging assembly task as even small errors can result in undesirable contacts. Recent efforts focused on Reinforcement Learning (RL), which often depends on careful definition of dense reward functions. This work proposes an effective strategy for such tasks that integrates traditional model-based control with RL to achieve improved insertion accuracy. The policy is trained exclusively in simulation and is zero-shot transferred to the real system. It employs a potential field-based controller to acquire a model-based policy for inserting a plug into a socket given full observability in simulation. This policy is then integrated with residual RL, which is trained in simulation given only a sparse, goal-reaching reward. A curriculum scheme over observation noise and action magnitude is used for training the residual RL policy. Both policy components use as input the SE(3) poses of both the plug and the socket and return the plug's SE(3) pose transform, which is executed by a robotic arm using a controller. The integrated policy is deployed on the real system without further training or fine-tuning, given a visual SE(3) object tracker. The proposed solution and alternatives are evaluated across a variety of objects and conditions in simulation and reality. The proposed approach outperforms recent RL-based methods in this domain and prior efforts with hybrid policies. Ablations highlight the impact of each component of the approach.

Figures (9)

• Figure 1: Zero-shot transfer of the policy learned in simulation to a real forceful insertion of an unseen plug and socket.
• Figure 2: From left to right: (a) A model-based policy is defined that generates a vector field under full observability. (b-c) An RL policy is trained in simulation given noisy pose observations to provide a residual action that is added to the output of the model-based policy. A sparse reward is provided only upon successful insertion (d) The final policy $\pi$ is zero-shot transferred to the real world, where observations come from a pose tracking module given RGB-D data. A controller translates the policy into robot joint controls.
• Figure 3: (left) The attractive potential field moves the object towards a nominal, straight-line insertion trajectory that leads to the goal pose; (right) the repulsive component pushes the object away from making contact with the socket, only when the object is close to it.
• Figure 4: Impact of observation noise on insertion trajectories: (left) Under no noise, both the model-based controller with and without the residual policy succeed. Residual RL helps to shorten trajectories. (center) With low observation noise, the performance of the model-based controller declines, but in combination with the residual policy output, the performance is preserved. (right) At high levels of noise, the model-based controller fails, while integrating the residual RL policy effectively compensates for the noisy pose estimate.
• Figure 5: Sim2Real policy transfer with 2 known (top) and 2 unknown at training objects (bottom).