Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution

TL;DR

This work formalizes Open-Set Domain Adaptation under background distribution shift and introduces CoLOR, a scalable constrained-learning solution with finite-sample guarantees. It provides a theoretical and empirical comparison against domain-discriminator methods, showing CoLOR's superior novelty detection and overall OSDA robustness, especially when the novel class is rare. The approach leverages multiple novelty heads and a shared representation to jointly optimize known-class accuracy and novel-class recall, validated across image and text benchmarks with varying α. The results offer practical guidance for deploying robust OSDA systems in real-world, shift-prone environments and point to avenues for integrating stronger theoretical guarantees with large-scale foundation-model features.

Abstract

As we deploy machine learning systems in the real world, a core challenge is to maintain a model that is performant even as the data shifts. Such shifts can take many forms: new classes may emerge that were absent during training, a problem known as open-set recognition, and the distribution of known categories may change. Guarantees on open-set recognition are mostly derived under the assumption that the distribution of known classes, which we call the background distribution, is fixed. In this paper we develop CoLOR, a method that is guaranteed to solve open-set recognition even in the challenging case where the background distribution shifts. We prove that the method works under benign assumptions that the novel class is separable from the non-novel classes, and provide theoretical guarantees that it outperforms a representative baseline in a simplified overparameterized setting. We develop techniques to make CoLOR scalable and robust, and perform comprehensive empirical evaluations on image and text data. The results show that CoLOR significantly outperforms existing open-set recognition methods under background shift. Moreover, we provide new insights into how factors such as the size of the novel class influences performance, an aspect that has not been extensively explored in prior work.

Figures (4)

• Figure 1: An instantiation of OSDA with background shift in addition to a novel subgroup. Top: Source dataset with $P_{{\mathcal{S}}}(x|y)$ representing distribution of subclasses of the labeled $k$ known categories ($y\in[k])$. Bottom: Target dataset with $P_{{\mathcal{T}}}(x|y)$ representing the distribution of subclasses of unlabeled known classes ($y\in[k]$) along with a novel subgroup ($y=k+1$). Unlike pure label shift or covariate shift, both $P_{{\mathcal{S}}}(x)$ or $P_{{\mathcal{S}}}(y)$ may change to $P_{{\mathcal{T}}}(x)$ and $P_{{\mathcal{T}}}(y)$ under background distribution shift.
• Figure 2: (left) CoLOR architecture for OSDA, heads $w^a_i$ for multiple recall values and classification heads $w^c$, vs. (right) a network optimizing for novelty detection with single recall value as in wald2023birds.
• Figure 3: (a) OSDA performance of top performing adaptive methods on SUN397 dataset with background shift using pretrained ResNet50 & CLIP ViT-L/14 backbone architectures. (b) Impact of novel class ratio ($\alpha$) on adaptive methods on SUN397 dataset under background shift.
• Figure 4: Effects of varying the FPR threshold on CoLOR method performance on SUN397 dataset.

Theorems & Definitions (17)

• Definition 1
• Lemma 1
• Definition 2
• Theorem 1
• proof : Proof sketch
• Lemma 2
• proof
• Proposition 1
• proof
• Lemma 3