OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation

TL;DR

This work proposes the task of open-vocabulary domain adaptation to infuse domain-specific knowledge into VLMs while preserving their open-vocabulary nature, and is the only parameter-efficient method that consistently surpasses the original VLM on novel classes.

Abstract

Recently, Vision-Language Models (VLMs) have advanced segmentation techniques by shifting from the traditional segmentation of a closed-set of predefined object classes to open-vocabulary segmentation (OVS), allowing users to segment novel classes and concepts unseen during training of the segmentation model. However, this flexibility comes with a trade-off: fully-supervised closed-set methods still outperform OVS methods on base classes, that is on classes on which they have been explicitly trained. This is due to the lack of pixel-aligned training masks for VLMs (which are trained on image-caption pairs), and the absence of domain-specific knowledge, such as autonomous driving. Therefore, we propose the task of open-vocabulary domain adaptation to infuse domain-specific knowledge into VLMs while preserving their open-vocabulary nature. By doing so, we achieve improved performance in base and novel classes. Existing VLM adaptation methods improve performance on base (training) queries, but fail to fully preserve the open-set capabilities of VLMs on novel queries. To address this shortcoming, we combine parameter-efficient prompt tuning with a triplet-loss-based training strategy that uses auxiliary negative queries. Notably, our approach is the only parameter-efficient method that consistently surpasses the original VLM on novel classes. Our adapted VLMs can seamlessly be integrated into existing OVS pipelines, e.g., improving OVSeg by +6.0% mIoU on ADE20K for open-vocabulary 2D segmentation, and OpenMask3D by +4.1% AP on ScanNet++ Offices for open-vocabulary 3D instance segmentation without other changes. The project page is available at https://open-das.github.io/.

Figures (15)

• Figure 1: Open-Vocabulary Domain Adaptation for Segmentation. We adapt VLMs to new domains while preserving their open-vocabulary nature, and integrate them to existing OVS pipelines such as OVSeg liang2023_ovseg(top) and OpenMask3D takmaz2023openmask3d(bottom). We show the results with a seen query "road" for 2D and the similarity score with an unseen query "tripod" for 3D, with red indicating high similarity.
• Figure 2: Illustration of the OpenDAS architecture. Left: Our work builds on CLIP radford2021learning, a VLM pre-trained on image-caption pairs with a contrastive loss $\mathcal{L}_c$. Center: We adapt the CLIP text- and image-encoders using prompt tuning with base (training) queries and generated negative queries to inject domain-specific priors. We insert visual prompts, $\mathbf{p}_{v}^{(0)}, ..., \mathbf{p}_{v}^{(J-1)}$, and textual prompts, $\mathbf{p}_{t}^{(0)}, ..., \mathbf{p}_{t}^{(J-1)}$, to the input of the encoder layers, $1, ..., J$. We combine cross-entropy loss $\mathcal{L}_{ce}$ with triplet loss $\mathcal{L}_t$ and negative queries to enhance CLIP's performance on novel (unseen) queries. Right: We integrate our model to existing OVS pipelines, i.e., OVSeg for 2D and OpenMask3D for 3D and test it with visually similar domains and novel queries, showing its open-vocabulary understanding capabilities while still adapting to the target domain.
• Figure 3: Supervised Domain Adaptation (SDA) vs. our Open-Vocabulary Domain Adaptation (OVDA). Supervised domain adaptation assumes the same vocabulary at training and test time, i.e., $Q_\text{train} = Q_\text{test} = Q_\text{base}$. We introduce open-vocabulary domain adaptation, where we expect the model to learn from training (base) queries, $Q_\text{train} = Q_\text{base}$, in the target domain and respond to unseen (novel) queries, $Q_\text{novel}$, at test time.
• Figure 4: Triplet Mining. We first instruct GPT-4 achiam2023gpt_4 to generate negative queries for a given set of base queries. During training, we feed the base queries and negative queries along with the image segments to the model. Then, we perform online hard negative sample mining, where we find the query with the minimum distance to the visual embedding of the corresponding segment.
• Figure 5: Qualitative Results on 2D Segment Classification. We show the predicted object classes with the ground truth masks given on three datasets.