Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection

Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tiruneh, Seungryul Baek

TL;DR

The paper tackles class incremental object detection (CIOD) by addressing catastrophic forgetting through a diffusion-based deep generative replay approach, SDDGR. It leverages a pre-trained Stable Diffusion model enhanced with grounding inputs (class labels and bounding boxes) to synthesize realistic old-class images, refined iteratively and filtered via dynamic thresholds. Training integrates an L2 distillation loss from the previous model and employs pseudo-labeling to prevent misclassifying old objects as background, achieving state-of-the-art results on COCO 2017 without real-data replay. The combination of grounding-guided generation, iterative refinement, pseudo-labeling, and distillation offers a practical pathway to robust CIOD with synthetic data, and the authors provide public code for reproducibility.

Abstract

In the field of class incremental learning (CIL), generative replay has become increasingly prominent as a method to mitigate the catastrophic forgetting, alongside the continuous improvements in generative models. However, its application in class incremental object detection (CIOD) has been significantly limited, primarily due to the complexities of scenes involving multiple labels. In this paper, we propose a novel approach called stable diffusion deep generative replay (SDDGR) for CIOD. Our method utilizes a diffusion-based generative model with pre-trained text-to-diffusion networks to generate realistic and diverse synthetic images. SDDGR incorporates an iterative refinement strategy to produce high-quality images encompassing old classes. Additionally, we adopt an L2 knowledge distillation technique to improve the retention of prior knowledge in synthetic images. Furthermore, our approach includes pseudo-labeling for old objects within new task images, preventing misclassification as background elements. Extensive experiments on the COCO 2017 dataset demonstrate that SDDGR significantly outperforms existing algorithms, achieving a new state-of-the-art in various CIOD scenarios. The source code will be made available to the public.

SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection

TL;DR

The paper tackles class incremental object detection (CIOD) by addressing catastrophic forgetting through a diffusion-based deep generative replay approach, SDDGR. It leverages a pre-trained Stable Diffusion model enhanced with grounding inputs (class labels and bounding boxes) to synthesize realistic old-class images, refined iteratively and filtered via dynamic thresholds. Training integrates an L2 distillation loss from the previous model and employs pseudo-labeling to prevent misclassifying old objects as background, achieving state-of-the-art results on COCO 2017 without real-data replay. The combination of grounding-guided generation, iterative refinement, pseudo-labeling, and distillation offers a practical pathway to robust CIOD with synthetic data, and the authors provide public code for reproducibility.

Abstract

In the field of class incremental learning (CIL), generative replay has become increasingly prominent as a method to mitigate the catastrophic forgetting, alongside the continuous improvements in generative models. However, its application in class incremental object detection (CIOD) has been significantly limited, primarily due to the complexities of scenes involving multiple labels. In this paper, we propose a novel approach called stable diffusion deep generative replay (SDDGR) for CIOD. Our method utilizes a diffusion-based generative model with pre-trained text-to-diffusion networks to generate realistic and diverse synthetic images. SDDGR incorporates an iterative refinement strategy to produce high-quality images encompassing old classes. Additionally, we adopt an L2 knowledge distillation technique to improve the retention of prior knowledge in synthetic images. Furthermore, our approach includes pseudo-labeling for old objects within new task images, preventing misclassification as background elements. Extensive experiments on the COCO 2017 dataset demonstrate that SDDGR significantly outperforms existing algorithms, achieving a new state-of-the-art in various CIOD scenarios. The source code will be made available to the public.
Paper Structure (18 sections, 6 equations, 4 figures, 7 tables)

This paper contains 18 sections, 6 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Continual learning
  4. Diffusion models
  5. Preliminaries
  6. Stable diffusion
  7. Controllable image generation
  8. Methods
  9. Image generation
  10. Iterative class-wise refiner
  11. Pseudo labeling
  12. Training with generated image
  13. Experiments
  14. Dataset and metrics
  15. Implementation and experiments
  16. ...and 3 more sections

Figures (4)

  • Figure 1: We utilize a pre-trained text-to-image diffusion model rombach2022high to generate realistic images that include objects from the old task. These images are then filtered out via iterative refinement and filtered synthetic images are integrated into the training process of the new task. During training, we employ L2 distillation to a synthetic dataset. Additionally, when training an image for the new task, we employ a pseudo-labeling that finds the old task objects from the new task images. The series of methods enable us to effectively mitigate the issue of catastrophic forgetting.
  • Figure 2: Schematic of Our SDDGR Framework: In the 'Generation process', our method individually generates each image based on class labels $\mathbf{C}_\text{label}$, specific bounding box locations $\mathbf{B}_\text{location}$, and old real images $\mathbf{x}_{m-1}$ in the old dataset $\mathcal{D}_{m-1}$. An 'Iterative refinement', employing the trained model $\mathcal{M}_{m-1}$, is applied to these synthetic images. In this algorithm, images with object scores below a dynamically adjusted threshold (ranging from 0.8 to 0.4 in our study) are systematically excluded. This cycle of generation and dynamic refinement continues until each class reaches the pre-defined target number of instances $\mathcal{N}$, or the lower threshold limit is met. In the 'Training process', the synthetic dataset is utilized for the continual learning via L2 distillation loss. Furthermore, real images undergo pseudo-labeling before being incorporated into the 'Training process'.
  • Figure 3: Differences in image generation based on input types. Each row represents different examples used for image synthesis. The first row uses prompts like "A photo of two umbrellas, person and boat, realistic, ... details". The second row uses prompts like "A photo of two elephants and person, ...". The last row uses prompts like "A photo of two suitcases, ...". (a) and (b) show the grounding input. (c) shows COCO real images. (d) depicts the prompt-only synthetic images. (e) depicts combined the grounding input and the prompt. (f) shows used the prompt, grounding input, and CLIP image embedding for image synthesis.
  • Figure 4: Class-specific image generation. This process utilizes a single class label as a prompt and grounding input with a fixed location. For example, in the first column, we have $\mathbf{T}$ = "stop sign", $\{\mathbf{C}_\text{label}$, $\mathbf{B}_\text{location}\}$ = {("stop sign", [0.3, 0.3, 0.6, 0.6])}.