PACF: Prototype Augmented Compact Features for Improving Domain Adaptive Object Detection

TL;DR

Domain Adaptive Object Detection suffers from a domain gap that broadens intra-class variance and shifts class means in the target domain. PACF tackles this by combining a prototype cross entropy loss, which aligns target RoI features to both source and target class prototypes, with a mutual regularization strategy that balances feature compactness and discriminability between linear and prototype-based classifiers; a theoretical lower bound on the target feature likelihood underpins the approach. The method uses a teacher–student framework with prototype-based classifiers and dynamically updated class prototypes, leading to a more compact and discriminative cross-domain feature space. Empirical results across Cityscapes↔Foggy Cityscapes, Sim10k↔Cityscapes, and KITTI↔Cityscapes show consistent gains over strong baselines, validating the effectiveness and generality of PACF for robust cross-domain object detection.

Abstract

In recent years, there has been significant advancement in object detection. However, applying off-the-shelf detectors to a new domain leads to significant performance drop, caused by the domain gap. These detectors exhibit higher-variance class-conditional distributions in the target domain than that in the source domain, along with mean shift. To address this problem, we propose the Prototype Augmented Compact Features (PACF) framework to regularize the distribution of intra-class features. Specifically, we provide an in-depth theoretical analysis on the lower bound of the target features-related likelihood and derive the prototype cross entropy loss to further calibrate the distribution of target RoI features. Furthermore, a mutual regularization strategy is designed to enable the linear and prototype-based classifiers to learn from each other, promoting feature compactness while enhancing discriminability. Thanks to this PACF framework, we have obtained a more compact cross-domain feature space, within which the variance of the target features' class-conditional distributions has significantly decreased, and the class-mean shift between the two domains has also been further reduced. The results on different adaptation settings are state-of-the-art, which demonstrate the board applicability and effectiveness of the proposed approach.

Figures (8)

• Figure 1: When a detector trained on the source is directly applied to target images, the domain gap causes a shift in feature distribution. We observe that intra-class conditional distribution which is compact on the source tends to scatter on target images, where target feature distribution exhibits mean shift and larger intra-class variance compared with the source.
• Figure 2: The overview of the proposed Prototype Augmented Compact Features (PACF) framework for DAOD. The teacher model adopts the target images with weak augmentation and the predictions of the teacher model are filtered as pseudo labels. Simultaneously, the source and target images with strong augmentation are fed into the student model. The prototype cross entropy loss $L_{pce}$ encourages target features to simultaneously move closer to the prototypes of both domains belonging to the same class, while staying far from prototypes of other classes. In addition, a mutual regularization loss $L_{mut}$ is proposed to balance feature compactness and discriminability on the target domain.
• Figure 3: (a) Illustration of Prototype-based Classifier and Mutual Regularization. Source and target prototypes are utilized to classifier target ROI features, and their classifier results are conducted mutual regularization loss with linear classifier. (b) The effect of PCMR framework for source and target prototypes.
• Figure 4: t-SNE visualization of target RoI features in the Cityscapes$\rightarrow$FoggyCityscapes setting.
• Figure 5: The correlation between linear classification scores and prototype-based cosine similarity. Left and right represent without and with mutual regularization.