Domain Adaptation for Large-Vocabulary Object Detectors

TL;DR

The paper tackles the domain shift and vocabulary mismatch that hinder large-vocabulary object detectors (LVDs) when deployed on diverse downstream data. It proposes Knowledge Graph Distillation (KGD), which leverages CLIP's implicit knowledge graphs through a two-stage process: Knowledge Graph Extraction (KGExtract) to build explicit language and vision graphs from downstream data, and Knowledge Graph Encapsulation (KGEncap) to inject these graphs into LVDs for improved cross-domain classification and localization. KGD comprises Language KG Distillation (LKG) and Vision KG Distillation (VKG), enabling complementary textual and visual semantic transfer via a two-layer GCN and a dynamic VKG with graph diffusion, respectively; the final pseudo labels are a normalized fusion of LKG- and VKG-guided predictions. Across 11 datasets, KGD consistently outperforms state-of-the-art unsupervised domain adaptation methods, demonstrating the effectiveness of explicit CLIP knowledge graphs for robust cross-domain detection and highlighting the practical impact of multi-modal, graph-based knowledge transfer for LVDs.

Abstract

Large-vocabulary object detectors (LVDs) aim to detect objects of many categories, which learn super objectness features and can locate objects accurately while applied to various downstream data. However, LVDs often struggle in recognizing the located objects due to domain discrepancy in data distribution and object vocabulary. At the other end, recent vision-language foundation models such as CLIP demonstrate superior open-vocabulary recognition capability. This paper presents KGD, a Knowledge Graph Distillation technique that exploits the implicit knowledge graphs (KG) in CLIP for effectively adapting LVDs to various downstream domains. KGD consists of two consecutive stages: 1) KG extraction that employs CLIP to encode downstream domain data as nodes and their feature distances as edges, constructing KG that inherits the rich semantic relations in CLIP explicitly; and 2) KG encapsulation that transfers the extracted KG into LVDs to enable accurate cross-domain object classification. In addition, KGD can extract both visual and textual KG independently, providing complementary vision and language knowledge for object localization and object classification in detection tasks over various downstream domains. Experiments over multiple widely adopted detection benchmarks show that KGD outperforms the state-of-the-art consistently by large margins.

Figures (3)

• Figure 1: A comparison of the domain adaptation performance of our method against existing methods. Our method outperforms the state-of-the-art consistently on 11 widely studied downstream detection datasets in terms of AP50 improvements. The results of all methods are acquired with the same baseline zhou2022detecting.
• Figure 2: Overview of the proposed Knowledge Graph Distillation (KGD). KGD comprises two consecutive stages including Knowledge Graph Extraction (KGExtract) and Knowledge Graph Encapsulation (KGEncap). KGExtract employs CLIP to encode downstream data as nodes and considers their feature distances as edges, explicitly constructing KGs including language knowledge graph (LKG) and vision knowledge graph (VKG) that inherit the rich semantic relations in CLIP. KGEncap transfers the extracted KGs into the large-vocabulary object detector to enable accurate object classification over downstream data. Besides, KGD works for both image and text data and allow extracting and transferring vision KG (VKG) and language KG (LKG), providing complementary knowledge for adapting large-vocabulary object detectors for handling various unlabelled downstream domains.
• Figure 3: Qualitative comparisons over various datasets. Zoom in for details.