Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization

TL;DR

The paper tackles single-domain generalization by proposing a dynamic object-centric learning framework that leverages prompt-guided gating to adapt to scene complexity. It introduces a prompt-based object-centric gating module, built on Slot Attention and CLIP prompts, coupled with a dynamic selective module to activate spatial and channel features within the backbone. The approach improves generalization on image classification and object detection across diverse target domains, outperforming state-of-the-art baselines and demonstrating robustness to varying scene conditions. By integrating multimodal fusion, object-centric representations, and sparsity-aware training, the method provides a practical mechanism to cope with domain shifts without requiring target-domain data.

Abstract

Single-domain generalization aims to learn a model from single source domain data to achieve generalized performance on other unseen target domains. Existing works primarily focus on improving the generalization ability of static networks. However, static networks are unable to dynamically adapt to the diverse variations in different image scenes, leading to limited generalization capability. Different scenes exhibit varying levels of complexity, and the complexity of images further varies significantly in cross-domain scenarios. In this paper, we propose a dynamic object-centric perception network based on prompt learning, aiming to adapt to the variations in image complexity. Specifically, we propose an object-centric gating module based on prompt learning to focus attention on the object-centric features guided by the various scene prompts. Then, with the object-centric gating masks, the dynamic selective module dynamically selects highly correlated feature regions in both spatial and channel dimensions enabling the model to adaptively perceive object-centric relevant features, thereby enhancing the generalization capability. Extensive experiments were conducted on single-domain generalization tasks in image classification and object detection. The experimental results demonstrate that our approach outperforms state-of-the-art methods, which validates the effectiveness and generally of our proposed method.

Figures (5)

• Figure 1: Illustration of dynamic object-centric learning via prompts for single domain generalization. Object-centric features capture the essential information related to individual objects. Incorporating the given scene prompts to dynamically optimize the extraction of object-centric features is beneficial for improving the generalization performance of models.
• Figure 2: Illustration of our proposed prompt-based dynamic object-centric learning network for single domain generalization. This method mainly includes a prompt-based object-centric gating module and a dynamic selective module. First, the Slot Attention multimodal fusion module extracts object-centric features and leverages the various scene prompts to guide the object-centric gating mask learning for the input from different scenes. Next, the gating mask is used to dynamically select the relevant object-centric features to improve the generalization ability.
• Figure 3: Illustration of our proposed prompt-based object-centric gating module.
• Figure 4: The t-SNE of our method on the target domain of PACS. The upper left domain name is the source domain.
• Figure 5: Qualitative results of object detection on the urban scene Diverse-Weather Dataset. We visualized the detection results in the target domain, where the top row represents the detection results of CLIP-Gap vidit2023clip, and the bottom row corresponds to our proposed method. It can be observed that in the "Night-Clear" scene, our method achieves more accurate car detection compared to CLIP-Gap vidit2023clip. In the complex "Night-Rainy" scene, CLIP-Gap vidit2023clip fails to detect the person, whereas our method successfully detects the person. In the "Day-Foggy" scene, our method accurately detects small-sized buses. Furthermore, in the "Dusk-Rainy" scene, our method exhibits improved accuracy in identifying and localizing trucks.