Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection

TL;DR

This work tackles the challenge of increasing dataset diversity for object detection without sacrificing semantic coordination. It introduces a diffusion-based data augmentation framework with three components: a Category Affinity Matrix derived from CLIP-based embeddings to guide inter-class diversity, a Surrounding Region Alignment strategy to preserve global semantic coherence during object edits via DDIM inversion and text-conditioned editing, and an instance-level filter to ensure quality. Empirically, the method yields substantial improvements across Faster R-CNN, Mask R-CNN, and YOLOX on standard benchmarks, with average gains of +1.4AP, +0.9AP, and +3.4AP over competitive baselines, and notable gains on category-specific and fine-grained datasets. These results demonstrate that diffusion-based augmentation can simultaneously enhance diversity and semantic coordination, offering practical benefits for robust object detection across diverse data regimes.

Abstract

Recent studies emphasize the crucial role of data augmentation in enhancing the performance of object detection models. However,existing methodologies often struggle to effectively harmonize dataset diversity with semantic coordination.To bridge this gap, we introduce an innovative augmentation technique leveraging pre-trained conditional diffusion models to mediate this balance. Our approach encompasses the development of a Category Affinity Matrix, meticulously designed to enhance dataset diversity, and a Surrounding Region Alignment strategy, which ensures the preservation of semantic coordination in the augmented images. Extensive experimental evaluations confirm the efficacy of our method in enriching dataset diversity while seamlessly maintaining semantic coordination. Our method yields substantial average improvements of +1.4AP, +0.9AP, and +3.4AP over existing alternatives on three distinct object detection models, respectively.

Figures (8)

• Figure 1: Visual presentation of obtained images and experimental results. Our data augmentation method strikes a balance between semantic coordination and image diversity, leading to the highest performance improvement on AP50.
• Figure 2: Overview of our method. Step 1 constructs the Category Affinity Matrix to develop tailored augmentation strategies for enhancing data diversity in each image. In Step 2, the method generates diverse images under the guidance of the Strategy Selection Module while maintaining semantic coordination through Surrounding Region Alignment with a diffusion model. Step 3 involves the exclusion of low-quality images at the instance level to further ensure the overall dataset quality.
• Figure 3: The process of the Image Processing Module. $\textbf{(1)}$ We get the initial noise $\tilde{z}_t$ through DDIM inversion. $\textbf{(2)}$ We take the Surrounding Region Alignment in the environment region and conditional control editing in the object region.
• Figure 4: Average cosine similarity between the augmented and original images on subsets of COCO, Objects365, and Open images.
• Figure 4: The effectiveness of our proposed Matrix, strategy, and instance-level filter. Matrix and Alig. refer to Category Affinity Matrix and Surrounding Region Alignment.