Boosting Single-domain Generalized Object Detection via Vision-Language Knowledge Interaction

TL;DR

This work tackles Single-Domain Generalized Object Detection by introducing a cross-modal, region-aware learning framework that leverages vision-language knowledge to improve region-level invariance across unseen domains. The core mechanisms, Cross-modal and Region-aware Feature Interaction (CRFI) and Cross-domain Proposal Refining and Mixing (CPRM), enable fine-grained text-image interactions and cross-domain proposal alignment, respectively, with an objective that combines ROI loss, CPRM loss, and CRFI loss. Empirical results on Cityscapes-C and Diverse Weather Dataset (DWD) demonstrate state-of-the-art improvements in mPC (+8.8 on Cityscapes-C and +7.9 on DWD) over strong baselines, validating both robustness to corruptions and adaptability to diverse weather conditions. The study shows the practical potential of integrating region-level vision-language cues into domain-general detection, paving the way for more reliable multimedia perception systems in variable environments.

Abstract

Single-Domain Generalized Object Detection~(S-DGOD) aims to train an object detector on a single source domain while generalizing well to diverse unseen target domains, making it suitable for multimedia applications that involve various domain shifts, such as intelligent video surveillance and VR/AR technologies. With the success of large-scale Vision-Language Models, recent S-DGOD approaches exploit pre-trained vision-language knowledge to guide invariant feature learning across visual domains. However, the utilized knowledge remains at a coarse-grained level~(e.g., the textual description of adverse weather paired with the image) and serves as an implicit regularization for guidance, struggling to learn accurate region- and object-level features in varying domains. In this work, we propose a new cross-modal feature learning method, which can capture generalized and discriminative regional features for S-DGOD tasks. The core of our method is the mechanism of Cross-modal and Region-aware Feature Interaction, which simultaneously learns both inter-modal and intra-modal regional invariance through dynamic interactions between fine-grained textual and visual features. Moreover, we design a simple but effective strategy called Cross-domain Proposal Refining and Mixing, which aligns the position of region proposals across multiple domains and diversifies them, enhancing the localization ability of detectors in unseen scenarios. Our method achieves new state-of-the-art results on S-DGOD benchmark datasets, with improvements of +8.8\%~mPC on Cityscapes-C and +7.9\%~mPC on DWD over baselines, demonstrating its efficacy.

Figures (6)

• Figure 1: This figure provides a high-level comparison of three distinct strategies in S-DGOD. Different shapes in the figure mean different regions. (a) Vision-only methods; (b) VLM-based methods; (c) Our method.
• Figure 2: This figure illustrates our CRFI, which extracts multi-region invariant features through text-image interaction.
• Figure 3: This figure illustrates our CPRM, which enables RPN to increasingly localize domain-invariant proposals across domains as training proceeds.
• Figure 4: The visualizations of the baseline, PhysAug, and our model on the Cityscapes-C dataset are presented. These include results for the clean validation set, as well as specific corruptions such as shot noise, zoom blur, snow, and contrast. All images from the Cityscapes-C dataset are evaluated at a corruption level of 5.
• Figure 5: Visualization of Daytime Foggy in DWD.