Beyond Boundaries: Leveraging Vision Foundation Models for Source-Free Object Detection

TL;DR

This work tackles Source-Free Object Detection (SFOD), where a detector trained on a source domain must adapt to a target domain without access to source data. It introduces a Vision Foundation Model (VFM)–driven SFOD framework comprising Patch-weighted Global Feature Alignment (PGFA), Prototype-based Instance Feature Alignment (PIFA), and Dual-source Enhanced Pseudo-label Fusion (DEPF) to jointly improve transferability and discriminability. By distilling global VFM features, aligning instance-level semantics with VFM prototypes, and fusing predictions from VFMs and the teacher with entropy-aware weighting, the approach achieves state-of-the-art results across six benchmarks and three cross-domain settings. The findings demonstrate the value of external, large-scale visual knowledge for robust cross-domain detection and point to future directions in incorporating richer multimodal foundation models into SFOD and broader domain adaptation tasks.

Abstract

Source-Free Object Detection (SFOD) aims to adapt a source-pretrained object detector to a target domain without access to source data. However, existing SFOD methods predominantly rely on internal knowledge from the source model, which limits their capacity to generalize across domains and often results in biased pseudo-labels, thereby hindering both transferability and discriminability. In contrast, Vision Foundation Models (VFMs), pretrained on massive and diverse data, exhibit strong perception capabilities and broad generalization, yet their potential remains largely untapped in the SFOD setting. In this paper, we propose a novel SFOD framework that leverages VFMs as external knowledge sources to jointly enhance feature alignment and label quality. Specifically, we design three VFM-based modules: (1) Patch-weighted Global Feature Alignment (PGFA) distills global features from VFMs using patch-similarity-based weighting to enhance global feature transferability; (2) Prototype-based Instance Feature Alignment (PIFA) performs instance-level contrastive learning guided by momentum-updated VFM prototypes; and (3) Dual-source Enhanced Pseudo-label Fusion (DEPF) fuses predictions from detection VFMs and teacher models via an entropy-aware strategy to yield more reliable supervision. Extensive experiments on six benchmarks demonstrate that our method achieves state-of-the-art SFOD performance, validating the effectiveness of integrating VFMs to simultaneously improve transferability and discriminability.

Figures (9)

• Figure 1: Illustration of our VFM-enhanced SFOD motivation. Our method leverages general VFM knowledge in VFMs such as DINOv2 (visual encoder) and Grounding DINO (vision-language detector) to address pseudo-label bias and multi-scale feature misalignment, resulting in improved transferability and discriminability compared with previous self-training pipelines of SFOD.
• Figure 2: Overview of the proposed method. Unlabeled target images are fed into teacher and student models for self-training with detection loss. DEPF fuses teacher and Grounding DINO predictions to generate refined pseudo-labels, enhancing discriminability. PGFA and PIFA align multi-scale features from the student and VFM, leveraging the generic VFM space to improve transferability.
• Figure 3: PIFA and PGFA. PGFA adopts similarity-based patch weights for fine-grained global feature fusion. PIFA extracts instance feature from VFM to construct a momentum-updated prototype for contrastive alignment with student instance feature.
• Figure 4: Detection visualization on the Cityscapes-to-Foggy Cityscapes adaptation scenario among (a) Source only, (b) Mean Teacher, (c) DRU, (d) Ours, (e) Ground Truth. We zoom in the discriminative regions for each set of predictions.
• Figure 5: Hyperparameter sensitivity. $\mu$ is the momentum for prototype update, $\lambda$ is the loss balancing factor, EMA iteration is the number of iterations for every EMA update.