Dataset Enhancement with Instance-Level Augmentations

TL;DR

The concept of instance-level data augmentation by repainting parts of the image at the level of object instances is introduced and improves the performance and generalization of the state-of-the-art salient object detection, semantic segmentation and object detection models.

Abstract

We present a method for expanding a dataset by incorporating knowledge from the wide distribution of pre-trained latent diffusion models. Data augmentations typically incorporate inductive biases about the image formation process into the training (e.g. translation, scaling, colour changes, etc.). Here, we go beyond simple pixel transformations and introduce the concept of instance-level data augmentation by repainting parts of the image at the level of object instances. The method combines a conditional diffusion model with depth and edge maps control conditioning to seamlessly repaint individual objects inside the scene, being applicable to any segmentation or detection dataset. Used as a data augmentation method, it improves the performance and generalization of the state-of-the-art salient object detection, semantic segmentation and object detection models. By redrawing all privacy-sensitive instances (people, license plates, etc.), the method is also applicable for data anonymization. We also release fully synthetic and anonymized expansions for popular datasets: COCO, Pascal VOC and DUTS.

Figures (14)

• Figure 1: Overview. Given an image and ground truth (or predicted) segmentation mask, we estimate depth and edge maps at the image level. The annotation is decomposed into the per-object binary masks and class, which together form the conditioning of the inpaining model. We redraw every instance and recombine them into a final image using alpha-blending sorted by depth.
• Figure 2: Noise Accumulation. Images accumulate noise when repeatedly encoding and decoding to and from latent space. PSNR and SSIM compared to the original.
• Figure 3: Qualitative Evaluation. The examples of method performance on three different datasets. The method generalize well to the complex scenes and datasets with no ground truth instance labels.
• Figure 4: Data Anonymization. The method efficiently repaints each annotation in the complex scenes, strongly mitigating privacy concerns for sensitive instances such as people or cars.
• Figure 5: Inpainting Conditioning. Inpainting without conditioning (b) does not preserve the structure of the object. Depth conditioning alone (c) fails to generate finer details and sharp edges. The full model (d) accurately preserves original annotations while producing high-quality samples.