Erase, then Redraw: A Novel Data Augmentation Approach for Free Space Detection Using Diffusion Model

TL;DR

The paper addresses the limited semantic diversity in data augmentation for free space detection by proposing an Erase, then Redraw pipeline that uses SAM-based erasing of background objects followed by diffusion-model–driven inpainting conditioned on text prompts. This approach enables semantically varied redraws in erased regions, producing more diverse training data. Empirical results on KITTI road and Cityscapes show state-of-the-art or leading performance improvements across three architectures (U-Net, Swin-UNet, VM-UNet) as measured by F1-Score and IoU. The work highlights the practical impact of language-guided, diffusion-based augmentation for semantic segmentation tasks and suggests avenues for future enhancements like controllable generation and broader applicability.

Abstract

Data augmentation is one of the most common tools in deep learning, underpinning many recent advances including tasks such as classification, detection, and semantic segmentation. The standard approach to data augmentation involves simple transformations like rotation and flipping to generate new images. However, these new images often lack diversity along the main semantic dimensions within the data. Traditional data augmentation methods cannot alter high-level semantic attributes such as the presence of vehicles, trees, and buildings in a scene to enhance data diversity. In recent years, the rapid development of generative models has injected new vitality into the field of data augmentation. In this paper, we address the lack of diversity in data augmentation for road detection task by using a pre-trained text-to-image diffusion model to parameterize image-to-image transformations. Our method involves editing images using these diffusion models to change their semantics. In essence, we achieve this goal by erasing instances of real objects from the original dataset and generating new instances with similar semantics in the erased regions using the diffusion model, thereby expanding the original dataset. We evaluate our approach on the KITTI road dataset and achieve the best results compared to other data augmentation methods, which demonstrates the effectiveness of our proposed development.

Figures (4)

• Figure 1: A schematic diagram of our data augmentation method, it involves first erasing the region of interest within the background of the original image, and then re-drawing within the erased area to generate new synthetic data.
• Figure 2: The architecture of our proposed data augmentation pipeline. Our pipeline consists of two parts, namely, SAM-based erasing and stable-diffusion-based scene redrawing.
• Figure 3: The redrawing process of our method. After erasing the pixels of the region of interest, new data is generated through the reverse diffusion process of the well-trained diffusion model. Different text prompts can generate new image with different distributions. For example, in the figure, our textual prompts are "a traditional Chinese building” and "a traditional Arabic building”, resulting in the erased area producing buildings with completely different architectural styles.
• Figure 4: The comparison between the synthetic data generated by our method and the synthetic data generated by other data augmentation methods. The first row of images represents the original data from the KITTI road dataset fritsch2013new. The 2nd, 3rd, 4th, 5th, and 6th rows correspond to the synthetic data generated by the data augmentation methods RandomErasing zhong2020random, Cutout devries2017improved, Gridmask chen2020gridmask, CutMix yun2019cutmix, and our method, respectively.