Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Simple Background Augmentation Method for Object Detection with Diffusion Model

Yuhang Li, Xin Dong, Chen Chen, Weiming Zhuang, Lingjuan Lyu

TL;DR

The paper tackles data diversity limitations in object detection by proposing a simple, annotation-preserving background augmentation method using diffusion-based inpainting. By focusing on background augmentation rather than object augmentation and carefully controlling prompts, masks, and diffusion steps, the approach enhances robustness and generalization across COCO and PASCAL VOC, and across both CNN- and transformer-based architectures. Extensive ablations show that background augmentation yields consistent mAP gains, especially in low-data regimes, while object augmentation can be detrimental. The method integrates with standard training and semi-supervised frameworks, offering a practical, scalable path to more accurate and robust vision models.

Abstract

In computer vision, it is well-known that a lack of data diversity will impair model performance. In this study, we address the challenges of enhancing the dataset diversity problem in order to benefit various downstream tasks such as object detection and instance segmentation. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models, specifically text-to-image synthesis technologies like Stable Diffusion. Our method focuses on generating variations of labeled real images, utilizing generative object and background augmentation via inpainting to augment existing training data without the need for additional annotations. We find that background augmentation, in particular, significantly improves the models' robustness and generalization capabilities. We also investigate how to adjust the prompt and mask to ensure the generated content comply with the existing annotations. The efficacy of our augmentation techniques is validated through comprehensive evaluations of the COCO dataset and several other key object detection benchmarks, demonstrating notable enhancements in model performance across diverse scenarios. This approach offers a promising solution to the challenges of dataset enhancement, contributing to the development of more accurate and robust computer vision models.

A Simple Background Augmentation Method for Object Detection with Diffusion Model

TL;DR

The paper tackles data diversity limitations in object detection by proposing a simple, annotation-preserving background augmentation method using diffusion-based inpainting. By focusing on background augmentation rather than object augmentation and carefully controlling prompts, masks, and diffusion steps, the approach enhances robustness and generalization across COCO and PASCAL VOC, and across both CNN- and transformer-based architectures. Extensive ablations show that background augmentation yields consistent mAP gains, especially in low-data regimes, while object augmentation can be detrimental. The method integrates with standard training and semi-supervised frameworks, offering a practical, scalable path to more accurate and robust vision models.

Abstract

In computer vision, it is well-known that a lack of data diversity will impair model performance. In this study, we address the challenges of enhancing the dataset diversity problem in order to benefit various downstream tasks such as object detection and instance segmentation. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models, specifically text-to-image synthesis technologies like Stable Diffusion. Our method focuses on generating variations of labeled real images, utilizing generative object and background augmentation via inpainting to augment existing training data without the need for additional annotations. We find that background augmentation, in particular, significantly improves the models' robustness and generalization capabilities. We also investigate how to adjust the prompt and mask to ensure the generated content comply with the existing annotations. The efficacy of our augmentation techniques is validated through comprehensive evaluations of the COCO dataset and several other key object detection benchmarks, demonstrating notable enhancements in model performance across diverse scenarios. This approach offers a promising solution to the challenges of dataset enhancement, contributing to the development of more accurate and robust computer vision models.
Paper Structure (15 sections, 4 equations, 6 figures, 4 tables)

This paper contains 15 sections, 4 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Background Augmentation
  4. Generative Augmentation with Inpainting
  5. Object Augmentation or Background Augmentation?
  6. Utility-Aware Background Augmentation
  7. Training with Background Augmented Data
  8. Experimental Evaluation
  9. Visualization
  10. Object Detection Evaluation
  11. Comparisons on Transformer Architecture
  12. Ablation Studies
  13. Augmenting More Data
  14. Evaluation on PASCAL VOC
  15. Conclusion

Figures (6)

  • Figure 1: Examples of failed object augmentation in MS COCO dataset using stable diffusion v1-5.
  • Figure 2: Visualization of several failure cases for background augmentation. (a) using image caption as text prompt, (b) using our prompt method but the objects extend to the background.
  • Figure 3: Visualization of our background augmentation: (a) using mask erosion and (b) the results.
  • Figure 4: The overall background augmentation framework of our method.
  • Figure 5: Example images of our background augmentation on MS COCO dataset.
  • ...and 1 more figures