3D-VirtFusion: Synthetic 3D Data Augmentation through Generative Diffusion Models and Controllable Editing
Shichao Dong, Ze Yang, Guosheng Lin
TL;DR
This paper tackles the scarcity of labeled 3D training data for scene understanding by introducing 3D-VirtFusion, a zero-shot data augmentation pipeline that leverages foundation models to generate labeled 3D scenes without real data. The approach creates 2D object images using diffusion models guided by ChatGPT-generated structural prompts, enriches appearance via depth-conditioned ControlNet and texture prompts, applies automatic drag-based shape augmentation, reconstructs 3D objects with Wonder3D/NeuRIS, and composes scenes through a template-based stitching process that yields semantic and instance labels. Key contributions include a fully automatic, diverse 3D data generation pipeline, multiple layers of augmentation (structure, texture, and shape), and demonstrated improvements in 3D semantic segmentation on ScanNet v2 (+2.7 mIoU). The work reduces reliance on real data, enables scalable generation of labeled 3D content, and supports robust 3D perception under few-shot and long-tail scenarios, with practical implications for robotics, AR/VR, and autonomous systems.
Abstract
Data augmentation plays a crucial role in deep learning, enhancing the generalization and robustness of learning-based models. Standard approaches involve simple transformations like rotations and flips for generating extra data. However, these augmentations are limited by their initial dataset, lacking high-level diversity. Recently, large models such as language models and diffusion models have shown exceptional capabilities in perception and content generation. In this work, we propose a new paradigm to automatically generate 3D labeled training data by harnessing the power of pretrained large foundation models. For each target semantic class, we first generate 2D images of a single object in various structure and appearance via diffusion models and chatGPT generated text prompts. Beyond texture augmentation, we propose a method to automatically alter the shape of objects within 2D images. Subsequently, we transform these augmented images into 3D objects and construct virtual scenes by random composition. This method can automatically produce a substantial amount of 3D scene data without the need of real data, providing significant benefits in addressing few-shot learning challenges and mitigating long-tailed class imbalances. By providing a flexible augmentation approach, our work contributes to enhancing 3D data diversity and advancing model capabilities in scene understanding tasks.