Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guarantees

TL;DR

This work tackles multi-domain active learning by introducing Composite Active Learning (CAL), a two-level framework that jointly models domain similarity across domains and instance-level informativeness. CAL constructs surrogate domains from labeled data, learns a domain similarity structure, aligns feature spaces, and allocates labeling budgets across domains while performing per-domain instance-level queries; it is augmented with GraDS to enhance sample selection. The authors provide theoretical bounds showing that CAL achieves tighter error guarantees across all domains and prove that optimal budget shares align with domain similarity. Empirically, CAL delivers substantial gains over strong baselines on synthetic RotatingMNIST data and real-world multi-domain datasets, validating the approach and its scalability. The results highlight a practical path to cross-domain generalization under labeling constraints and open avenues for extensions to NLP and imbalanced-domain scenarios.

Abstract

Active learning (AL) aims to improve model performance within a fixed labeling budget by choosing the most informative data points to label. Existing AL focuses on the single-domain setting, where all data come from the same domain (e.g., the same dataset). However, many real-world tasks often involve multiple domains. For example, in visual recognition, it is often desirable to train an image classifier that works across different environments (e.g., different backgrounds), where images from each environment constitute one domain. Such a multi-domain AL setting is challenging for prior methods because they (1) ignore the similarity among different domains when assigning labeling budget and (2) fail to handle distribution shift of data across different domains. In this paper, we propose the first general method, dubbed composite active learning (CAL), for multi-domain AL. Our approach explicitly considers the domain-level and instance-level information in the problem; CAL first assigns domain-level budgets according to domain-level importance, which is estimated by optimizing an upper error bound that we develop; with the domain-level budgets, CAL then leverages a certain instance-level query strategy to select samples to label from each domain. Our theoretical analysis shows that our method achieves a better error bound compared to current AL methods. Our empirical results demonstrate that our approach significantly outperforms the state-of-the-art AL methods on both synthetic and real-world multi-domain datasets. Code is available at https://github.com/Wang-ML-Lab/multi-domain-active-learning.

Figures (6)

• Figure 1: Among the four domains, domain ${\mathcal{O}}_2$ is the closest to the other three domains. Assigning more labels to this representative domain ${\mathcal{O}}_2$ could better generalize to other domains, since similar domains may have similar decision boundaries.
• Figure 2: Overview of our proposed framework.Left:$3$ labeled domains and $3$ original domains in the input space. Right: In the latent space, CAL constructs surrogate domain ${\mathcal{S}}_2$ using $\alpha$-weighted labeled domains ${\mathcal{S}}_2(Z)=\sum\nolimits_{j=1}^3 \alpha_{2,j}{\mathcal{L}}_j(Z)$, and estimate similarity weights $\alpha_{2,j}$ by minimizing the distance between surrogate domain ${\mathcal{S}}_2$ and its original domain ${\mathcal{O}}_2$. All encoders $e$ share the same parameters. The encoder $e$, domain similarity $\{\alpha_{2,j}|j=1,2,3\}$, and conditional discriminator $f$ play a min-max game to reduce the distance between ${\mathcal{S}}_2$ and ${\mathcal{O}}_2$ by joint similarity estimation and feature alignment. The classifier $h$ is trained on all surrogate domains ${\mathcal{S}}_1$, ${\mathcal{S}}_2$ and ${\mathcal{S}}_3$. For clarity, we omit surrogate domains ${\mathcal{S}}_1$ and ${\mathcal{S}}_3$.
• Figure 3: Estimated similarity $\alpha_{i,j}$ and $\alpha_j$ for RotatingMNIST-D$6$.
• Figure 4: Samples in Datasets
• Figure 5: Visualization of Estimated $\hbox{\boldmath$\alpha$\unboldmath}$ for RotatingMNIST

Theorems & Definitions (11)

• Lemma 4.1: Error Bound for One Domain
• Theorem 4.1: Error Bound for All Domains
• Theorem 4.2: Optimal Budget Assignment
• Lemma B.1: Error Bound for One Domain
• proof
• Lemma B.2
• proof
• Theorem B.1: Error Bound for All Domains
• proof
• Theorem B.2: Optimal Budget Assignment