Object Style Diffusion for Generalized Object Detection in Urban Scene

TL;DR

GoDiff addresses the challenge of generalizing object detectors to unseen urban domains using a single labeled source. It introduces a two-part pipeline: Pseudo Target Data Generation (PTDG) using an InstanceDiffusion model to create diverse yet annotated pseudo-target images, and Cross Style Normalization (CSN) to align features across styles during training. The method adds an image-level dual-prompt strategy with object-level prompts, plus a CLIP-RBF object filtering and a covariance-based loss to promote style-invariant representations. Experiments on the Diverse Weather Dataset (DWD) and Cityscapes-C show consistent improvements over state-of-the-art baselines and demonstrate GoDiff's plug-and-play compatibility with other S-DG methods, including OA-DG. The work provides practical gains for autonomous driving scenarios by boosting robustness to weather and domain shifts, and the authors release code and generated data.

Abstract

Object detection is a critical task in computer vision, with applications in various domains such as autonomous driving and urban scene monitoring. However, deep learning-based approaches often demand large volumes of annotated data, which are costly and difficult to acquire, particularly in complex and unpredictable real-world environments. This dependency significantly hampers the generalization capability of existing object detection techniques. To address this issue, we introduce a novel single-domain object detection generalization method, named GoDiff, which leverages a pre-trained model to enhance generalization in unseen domains. Central to our approach is the Pseudo Target Data Generation (PTDG) module, which employs a latent diffusion model to generate pseudo-target domain data that preserves source domain characteristics while introducing stylistic variations. By integrating this pseudo data with source domain data, we diversify the training dataset. Furthermore, we introduce a cross-style instance normalization technique to blend style features from different domains generated by the PTDG module, thereby increasing the detector's robustness. Experimental results demonstrate that our method not only enhances the generalization ability of existing detectors but also functions as a plug-and-play enhancement for other single-domain generalization methods, achieving state-of-the-art performance in autonomous driving scenarios.

Figures (7)

• Figure 1: Our method expands the training data distribution through a two-step approach. First, LDMs generate a pseudo-target domain $D^{G}$ with diverse styles, partially covering multiple target domains $D^T$. This $D^{G}$ is combined with the source domain $D^S$ for model training. Second, during training, the data distribution is further augmented to maximize coverage of potential target domains, enhancing the model's generalization capability.
• Figure 2: Visualization of source domain and generated pseudo-target domains. The leftmost image shows a real Daytime-Sunny scene from the source domain. The four images on the right demonstrate our method's capability to generate diverse pseudo-target domain images while preserving the semantic content and annotations from the source domain.
• Figure 3: Pipeline for pseudo-target domain generation. The process takes annotated source domain images as input and produces annotated images for multiple pseudo-target domains. This approach enables the creation of diverse, style-rich images while preserving original annotations, facilitating domain generalization in object detection tasks.
• Figure 4: Cross Style Normalization-based detector training framework. The CSN modules are embedded between Backbone layers, processing feature maps from images with diverse styles. This approach leverages diverse image styles to create a more continuous feature distribution, promoting style-invariant feature learning for improved domain generalization.
• Figure 5: The Impact of source domain fine-tuning on generation domain authenticity (CMMD) and target domain performance (mAP).