GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detection

TL;DR

GenDet redefines object detection as image-conditioned generation by fine-tuning a pre-trained Stable Diffusion model to paint colored bounding boxes directly onto the input image, effectively modeling the conditional distribution $D(oldsymbol{y}|oldsymbol{x})$. It introduces a dual-path, multi-step diffusion framework with a gradient-preserving loss and both feature-based and learning-based post-processing to extract detections from generated outputs. The approach demonstrates competitive performance on COCO2017 and CrowdHuman, illustrating the viability of generation-based detection and the benefit of leveraging rich generative priors for structured visual understanding. This work points to a broader paradigm where discriminative tasks are solved within generative models, enabling unified pipelines for synthesis and perception with potential gains in flexibility and transferability.

Abstract

This paper presents GenDet, a novel framework that redefines object detection as an image generation task. In contrast to traditional approaches, GenDet adopts a pioneering approach by leveraging generative modeling: it conditions on the input image and directly generates bounding boxes with semantic annotations in the original image space. GenDet establishes a conditional generation architecture built upon the large-scale pre-trained Stable Diffusion model, formulating the detection task as semantic constraints within the latent space. It enables precise control over bounding box positions and category attributes, while preserving the flexibility of the generative model. This novel methodology effectively bridges the gap between generative models and discriminative tasks, providing a fresh perspective for constructing unified visual understanding systems. Systematic experiments demonstrate that GenDet achieves competitive accuracy compared to discriminative detectors, while retaining the flexibility characteristic of generative methods.

Figures (9)

• Figure 1: Difference between object detection methods. Existing object detectors (i.e., (a), (b), (c), and (d)) are discriminative methods, involving the classification of object categories and the regression of bounding box dimensions. In contrast, our GenDet takes an entirely different approach by formulating object detection as image generation task, directly rendering colored object bounding boxes onto the input image. This innovative strategy offers a more direct and intuitive way to perform object detection.
• Figure 2: Visualization of the generated object bounding boxes. GenDet is a diffusion-based image generation model designed for object detection. It utilizes the prior knowledge from Stable Diffusion to directly paint detection boxes on the original image, with different colors indicating different object categories. To minimize box overlap, the bounding boxes are scaled down, and red dots are added at the center of each object to enhance recognition accuracy.
• Figure 3: Overview of GenDet's training pipeline. Starting from a pre-trained Stable Diffusion model, both the image $x$ and its corresponding annotation image $y$ with colored bounding boxes are encoded through the pre-trained VAE. The noisy version of the annotation image, $z_{t}^y$, is obtained by introducing noise at a specific diffusion step $t \in [1, T]$. The U-Net unet input layer is modified to process the concatenated inputs, and the model is then fine-tuned with the standard diffusion objective, $v$-prediction v, following the multi-step training procedure. Additionally, the task prompt p is introduced to either generate annotation image $y$ or reconstruct the input image $x$.
• Figure 4: Illustration of different types of target images $\mathbf{y}$, which encode object detection bounding box information. To better align with the image generation process in Stable Diffusion ldm, we overlay bounding boxes on the original image, using distinct colors to differentiate object categories. To reduce overlap, as shown in (d), we shrink the bounding boxes and further enhance detection cues by marking the center of each box with a red dot.
• Figure 5: Overview of the GenDet inference scheme. Given an input image $\mathbf{x}$, GenDet begins by encoding it using the pre-trained Stable Diffusion VAE to generate the latent code $\mathbf{z}^x$. This latent code is then combined with the annotation image's latent $\mathbf{z}_t^\mathbf{y}$ and fed into the modified, fine-tuned U-Net unet at each denoising iteration. After completing $T$ steps of the diffusion process, the resulting latent $\mathbf{z}_0^\mathbf{y}$ is decoded into the prediction image $\hat{\mathbf{y}}$. The final object detection output is obtained by applying post-processing to $\hat{\mathbf{y}}$.