Towards Single-Source Domain Generalized Object Detection via Causal Visual Prompts

TL;DR

Single-source domain generalized object detection (SDGOD) suffers from spurious correlations learned in the source domain, hindering generalization to unseen targets. Cauvis introduces Cross-Attention Prompts that implement a back-door adjustment on causal features and a Dual-Branch Adapter comprising a Causal Branch and a Fourier-domain Auxiliary Branch to decouple causal from noncausal information; the backbone is frozen (DINOv2) to reduce training cost. The cross-attention prompts approximate $P(Y|do(X)) = \mathbb{E}_{Z}[P(Y|X,Z)]$ by spectral filtering that emphasizes top-$k$ causal directions, while the Fourier branch extracts high-frequency, domain-invariant features; together they yield substantial gains on SDGOD benchmarks, with improvements up to $+31.4$ in $mAP$ and enhanced robustness on Cityscapes-C and BDD100K-C. Overall, Cauvis advances SDGOD by fusing causal inference with prompt-based learning to deliver robust, single-source domain generalization for autonomous driving perception.

Abstract

Single-source Domain Generalized Object Detection (SDGOD), as a cutting-edge research topic in computer vision, aims to enhance model generalization capability in unseen target domains through single-source domain training. Current mainstream approaches attempt to mitigate domain discrepancies via data augmentation techniques. However, due to domain shift and limited domain-specific knowledge, models tend to fall into the pitfall of spurious correlations. This manifests as the model's over-reliance on simplistic classification features (e.g., color) rather than essential domain-invariant representations like object contours. To address this critical challenge, we propose the Cauvis (Causal Visual Prompts) method. First, we introduce a Cross-Attention Prompts module that mitigates bias from spurious features by integrating visual prompts with cross-attention. To address the inadequate domain knowledge coverage and spurious feature entanglement in visual prompts for single-domain generalization, we propose a dual-branch adapter that disentangles causal-spurious features while achieving domain adaptation via high-frequency feature extraction. Cauvis achieves state-of-the-art performance with 15.9-31.4% gains over existing domain generalization methods on SDGOD datasets, while exhibiting significant robustness advantages in complex interference environments.

Figures (5)

• Figure 1: The interference of spurious features (such as color) between the source domain and the target domain causes existing methods fasterrcnn to be prone to errors in the target domain, often mistakenly identifying the color white as trucks, showing poor generalization ability. Our method, leveraging Causal Visual Prompts combined with a branch that can extract high-frequency features, effectively stops the model from making predictions based on spurious features, improving its accuracy in the target domain. "Ice" signifies frozen parameters, while "fire" indicates fine-tunable parameters.
• Figure 2: Left: Visualization results of DINOv2 dinov2 and Cauvis, along with their heatmaps and the differences between. Right: (a) performance of Bus and Truck on the biased and unbiased test datasets. (b) Cauvis exhibits stronger activations both within the object’s interior and outside region.
• Figure 3: Overview of the Cauvis. The top part illustrates the Cross Attention mechanism and the Fourier Transformation. The bottom part shows the detailed structure of Cauvis, including Cross-Attention Prompts and Dual-Branch Adapter. The Cauvis module is integrated into each layer of the backbone. Blue for frozen parameters, orange for fine-tuning parameters.
• Figure 4: Left: Visualization results of four methods (SPT spt, Rein Rein, Cauvis (Ours)) in night scenes, where GT stands for Ground Truth. Right: Prompt length ablation under two strategies.
• Figure 5: More visualization of the detection results. The model is trained on Sunny, Foogy, Rainy scenes with DINOv2-L backbone.