End-to-end example-based sim-to-real RL policy transfer based on neural stylisation with application to robotic cutting

TL;DR

This work proposes a novel method for sim-to-real transfer of reinforcement learning policies, based on a reinterpretation of neural style transfer from image processing to synthesise novel training data from unpaired unlabelled real world datasets.

Abstract

Whereas reinforcement learning has been applied with success to a range of robotic control problems in complex, uncertain environments, reliance on extensive data - typically sourced from simulation environments - limits real-world deployment due to the domain gap between simulated and physical systems, coupled with limited real-world sample availability. We propose a novel method for sim-to-real transfer of reinforcement learning policies, based on a reinterpretation of neural style transfer from image processing to synthesise novel training data from unpaired unlabelled real world datasets. We employ a variational autoencoder to jointly learn self-supervised feature representations for style transfer and generate weakly paired source-target trajectories to improve physical realism of synthesised trajectories. We demonstrate the application of our approach based on the case study of robot cutting of unknown materials. Compared to baseline methods, including our previous work, CycleGAN, and conditional variational autoencoder-based time series translation, our approach achieves improved task completion time and behavioural stability with minimal real-world data. Our framework demonstrates robustness to geometric and material variation, and highlights the feasibility of policy adaptation in challenging contact-rich tasks where real-world reward information is unavailable.

Figures (10)

• Figure 1: Overview of proposed framework. In the first stage, a simulation of cutting mechanics is used to generate an expert policy and a variational autoencoder (VAE) is trained on simulated trajectory windows. In the second stage, the VAE encoded representations are used to generate pairings between a simulated and real world dataset which are used as style targets. Finally, expert trajectories are used to train a learner target domain policy with the generated observation windows.
• Figure 2: Overview of VAE encoder architecture; layer indices for style transfer are demarcated.
• Figure 2: Table of model parameters for cutting simulation (source domain)
• Figure 3: Effect of content-style weight ratio $w_{c}/w_{l}$ on normalised (unweighted) content-style loss, averaged over 5 content-style batches (batch size 256), with chosen value $w_{c}/w_{s}=0.02$ indicated (dotted line). Decreasing ratio results in diminishing returns on style while diverging substantially from the original content windows. Increasing ratio tends towards identity (generated windows correspond to unaltered simulated windows).
• Figure 4: Convergence plot for style transfer optimisation with parameters from Table \ref{['tab:Method-Hyperparameters']} for a batch of 256 content-style pairings.