Evaluating Text-to-Image Diffusion Models for Texturing Synthetic Data

TL;DR

Surprisingly, it is found that texturing using a diffusion model performs on par with random textures, despite generating seemingly more realistic images, suggesting that using diffusion models for texturing does not benefit synthetic data generation for robotics.

Abstract

Building generic robotic manipulation systems often requires large amounts of real-world data, which can be dificult to collect. Synthetic data generation offers a promising alternative, but limiting the sim-to-real gap requires significant engineering efforts. To reduce this engineering effort, we investigate the use of pretrained text-to-image diffusion models for texturing synthetic images and compare this approach with using random textures, a common domain randomization technique in synthetic data generation. We focus on generating object-centric representations, such as keypoints and segmentation masks, which are important for robotic manipulation and require precise annotations. We evaluate the efficacy of the texturing methods by training models on the synthetic data and measuring their performance on real-world datasets for three object categories: shoes, T-shirts, and mugs. Surprisingly, we find that texturing using a diffusion model performs on par with random textures, despite generating seemingly more realistic images. Our results suggest that, for now, using diffusion models for texturing does not benefit synthetic data generation for robotics. The code, data and trained models are available at \url{https://github.com/tlpss/diffusing-synthetic-data.git}.

Figures (4)

• Figure 1: Left: In this work, we compare text-to-image diffusion models against random textures for texturing 3D scenes in a synthetic data generation pipeline. Right: We evaluate the efficacy of the synthetic data on real-world data for both keypoint detection and segmentation.
• Figure 2: Scaling behavior of the different texturing approaches. For both diffusion textures and random textures, the performance improves with increasing data, though it starts to plateau around 5,000 images.
• Figure 3: Illustration of the one-stage approach issue: the background and object prompt are entangled in the visual output. The three-stage approach managed to separate them (though the table perspective is wrong).
• Figure 4: Comparison of average keypoint distances for different values of the Controlnet conditioning scale (CCS). The optimal value depends on the category, but 1.5 (marked in green) is a sensible default.