Language-Driven Dual Style Mixing for Single-Domain Generalized Object Detection

TL;DR

Language-Driven Dual Style Mixing (LDDS) is proposed for single-domain generalization, which diversifies the source domain by fully utilizing the semantic information of the VLM, allowing feature augmentation to be model-agnostic and can work seamlessly with the mainstream detector frameworks, including the one-stage, two-stage, and transformer-based detectors.

Abstract

Generalizing an object detector trained on a single domain to multiple unseen domains is a challenging task. Existing methods typically introduce image or feature augmentation to diversify the source domain to raise the robustness of the detector. Vision-Language Model (VLM)-based augmentation techniques have been proven to be effective, but they require that the detector's backbone has the same structure as the image encoder of VLM, limiting the detector framework selection. To address this problem, we propose Language-Driven Dual Style Mixing (LDDS) for single-domain generalization, which diversifies the source domain by fully utilizing the semantic information of the VLM. Specifically, we first construct prompts to transfer style semantics embedded in the VLM to an image translation network. This facilitates the generation of style diversified images with explicit semantic information. Then, we propose image-level style mixing between the diversified images and source domain images. This effectively mines the semantic information for image augmentation without relying on specific augmentation selections. Finally, we propose feature-level style mixing in a double-pipeline manner, allowing feature augmentation to be model-agnostic and can work seamlessly with the mainstream detector frameworks, including the one-stage, two-stage, and transformer-based detectors. Extensive experiments demonstrate the effectiveness of our approach across various benchmark datasets, including real to cartoon and normal to adverse weather tasks. The source code and pre-trained models will be publicly available at https://github.com/qinhongda8/LDDS.

Figures (12)

• Figure 1: Existing VLM-based methods vidit2023clipfahes2023poda (a) require that the detector's backbone has the same architecture as the image encoder, resulting in the limitation of detector framework selection. Besides, they only perform feature augmentation of the detector’s backbone, can not effectively transfer the semantic information embedded in the VLM. Our method (b) avoids these limitations with proposed dual-style mixing schemes for both image and feature augmentation.
• Figure 2: The pipeline of LDDS, which consists of style generation, image-level and smooth feature-level style mixing. We first feed the textual embeddings of the style prompts obtained from the CLIP together with the source image $I^s$ into the StyleNet to generate style diversified images ${I}^{st}$. Then, the source image ${I}^{s}$ and ${I}^{st}$ undergo image-level style mixing in the frequency domain to produce a style mixed image $I^{m}$. We perform a style statistics smooth operation on the statistical style data of $\mathbf{f}^{st}$ to avoid the possible style conflicts. Finally, the feature-level style mixing is performed with the statistical style data of the source image feature map $\mathbf{f}^m$. The resulting dual mixed style feature map $\mathbf{f}^{m\star}$ serves as fully supervised data for the detector.
• Figure 3: The visual results of the style generation and image-level style mixing, where the first and second columns correspond to the ${I}^{s}$ and the ${I}^{st}$, respectively. In the third column of ${I}^m$, while global style information has been mixed, the local styles, such as the object details and background textures, remain unchanged.
• Figure 4: The semantic information space representation of $\mathbf{f}^m$ and $\mathbf{f}^{st}$ (source content information in $\mathbf{f}^{m}$ is complete, while part of it is missing in $\mathbf{f}^{st}$). Direct feature augmentation can cause global style mixing conflicts, which in turn disrupts the global style information in $\mathbf{f}^{m}$.
• Figure 5: Qualitative results on the real to cartoon. From left to right, every two columns represent the samples of Clipart1k, Watercolor2k, and Comic2k, respectively. Our LDDS demonstrates better localization and classification of cartoon objects compared to the benchmark detector YOLOv8 yolov8.