Downstream-Pretext Domain Knowledge Traceback for Active Learning

TL;DR

DOKT introduces a principled way to fuse pretext-based representations with downstream domain knowledge for active learning by coupling a traceback diversity indicator with a domain-based uncertainty estimator. The traceback module connects low-level pretext space and high-level downstream space to quantify unlabeled data diversity, while the domain uncertainty estimator uses perceptual domain mixing and a ranking-based uncertainty objective to target informative samples near the decision boundary. Across ten datasets and multiple tasks, DOKT consistently outperforms state-of-the-art AL methods and shows strong generalization to semantic segmentation and image captioning, particularly under data scarcity and long-tailed conditions. The work demonstrates that integrating downstream knowledge with pretraining guidance yields more informative sample selection and improved labeling efficiency, with potential for extending to large pre-trained models and transfer learning scenarios.

Abstract

Active learning (AL) is designed to construct a high-quality labeled dataset by iteratively selecting the most informative samples. Such sampling heavily relies on data representation, while recently pre-training is popular for robust feature learning. However, as pre-training utilizes low-level pretext tasks that lack annotation, directly using pre-trained representation in AL is inadequate for determining the sampling score. To address this problem, we propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance for selecting diverse and instructive samples near the decision boundary. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. The diversity indicator constructs two feature spaces based on the pre-training pretext model and the downstream knowledge from annotation, by which it locates the neighbors of unlabeled data from the downstream space in the pretext space to explore the interaction of samples. With this mechanism, DOKT unifies the data relations of low-level and high-level representations to estimate traceback diversity. Next, in the uncertainty estimator, domain mixing is designed to enforce perceptual perturbing to unlabeled samples with similar visual patches in the pretext space. Then the divergence of perturbed samples is measured to estimate the domain uncertainty. As a result, DOKT selects the most diverse and important samples based on these two modules. The experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods and generalizes well to various application scenarios such as semantic segmentation and image captioning.

Figures (6)

• Figure 1: A pool-based AL cycle. In each iteration, the AL model is trained with labeled data. After training, a subset of unlabeled samples is selected based on the model inference and then labeled by an oracle. We leverage pre-training guidance and downstream knowledge to evaluate sampling score for better AL sampling. This AL cycle repeats until the model performance meets the user's requirements or the label budget runs out.
• Figure 2: The network architecture of proposed DOKT. It consists of a traceback diversity indicator and a domain uncertainty estimator. The indicator traces similar unlabeled samples in two spaces for traceback diversity. The estimator learns to predict the uncertainty score based on domain mixing samples and apply perceptual perturbing to jointly estimate the divergences of augmented samples for domain uncertainty. Finally, DOKT selects the most diverse and instructive samples based on the two modules. The red lines denote pipelines for unlabeled data flow and the blue lines for labeled data flow.
• Figure 3: Results of different AL methods on the image classification datasets. (a) and (b) are the results on CIFAR-10 and CIFAR-100. (c) is the result on SVHN. (d) is the result on the fine-grained dataset Aircraft. The dotted line shows the required labeled data to achieve a competitive performance and the required annotation of AL methods is detailed in Table \ref{['tabnum']}.
• Figure 4: Results of different AL methods. (a) and (b) are the results on long-tailed CIFAR-10 and CIFAR-100. (c) is the result on ImageNet. (d) is the result on iNaturalist.
• Figure 5: Results of different AL methods on semantic segmentation and image captioning datasets. (a) and (b) are the results on Cityscapes and BraTS. (c) and (d) are the results on MS-COCO using two metrics BLEU4 and CIDEr.