Sim2Real Instance-Level Style Transfer for 6D Pose Estimation

TL;DR

A simulation to reality (sim2real) instance-level style transfer for 6D pose estimation network training that transfers the style of target objects individually, from synthetic to real, without human intervention improves the quality of synthetic data for training pose estimation networks.

Abstract

In recent years, synthetic data has been widely used in the training of 6D pose estimation networks, in part because it automatically provides perfect annotation at low cost. However, there are still non-trivial domain gaps, such as differences in textures/materials, between synthetic and real data. These gaps have a measurable impact on performance. To solve this problem, we introduce a simulation to reality (sim2real) instance-level style transfer for 6D pose estimation network training. Our approach transfers the style of target objects individually, from synthetic to real, without human intervention. This improves the quality of synthetic data for training pose estimation networks. We also propose a complete pipeline from data collection to the training of a pose estimation network and conduct extensive evaluation on a real-world robotic platform. Our evaluation shows significant improvement achieved by our method in both pose estimation performance and the realism of images adapted by the style transfer.

Figures (10)

• Figure 1: Top: Training of the style transfer network using weakly-paired images between synthetic and real. Bottom: Style transfer from newly generated synthetic images to real ones, using the style transfer network.
• Figure 2: The workflow of the proposed domain adaptation and training method: (1) Weakly-paired images are generated by using a robot and a 6D pose estimation network trained on synthetic images only. To generate these images properly, a mismatch filter is utilized as described in \ref{['weakly-paired']}. (2) Then, a style transfer network is trained based on the weakly-paired images, and convert synthetic images via instance-level style transfer. (3) Lastly, the 6D pose estimation network is trained using the adapted synthetic images.
• Figure 3: Training of an instance-level style transfer network on weakly-paired images. $G$ represents the style-mapping function from synthetic to real. $D$ represents the discriminator for synthetic and real.
• Figure 4: The process of individual object style transfer
• Figure 5: Average distance threshold curves of non-adaptation, our adaptation method, and other methods including CycleGAN Zhu2017-mq, DRIT lee2020drit++, CUT Park2020-lf.