ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learning

TL;DR

ADROIT addresses the high labeling cost in active learning by learning robust, task-aware representations from both labeled and unlabeled data. It unifies representation learning via a $eta$-VAE, enhances it with self-supervised rotation losses, explicit conditional modeling through a proxy task-learner, and a teacher–student distillation mechanism, combined with an adversarial discriminator to select informative unlabeled samples. The approach achieves superior performance on balanced and imbalanced datasets compared to state-of-the-art AL baselines, with ablation analyses confirming the contribution of each component. This framework reduces reliance on the labeling oracle and demonstrates practical gains for efficient active learning.

Abstract

Active learning aims to select optimal samples for labeling, minimizing annotation costs. This paper introduces a unified representation learning framework tailored for active learning with task awareness. It integrates diverse sources, comprising reconstruction, adversarial, self-supervised, knowledge-distillation, and classification losses into a unified VAE-based ADROIT approach. The proposed approach comprises three key components - a unified representation generator (VAE), a state discriminator, and a (proxy) task-learner or classifier. ADROIT learns a latent code using both labeled and unlabeled data, incorporating task-awareness by leveraging labeled data with the proxy classifier. Unlike previous approaches, the proxy classifier additionally employs a self-supervised loss on unlabeled data and utilizes knowledge distillation to align with the target task-learner. The state discriminator distinguishes between labeled and unlabeled data, facilitating the selection of informative unlabeled samples. The dynamic interaction between VAE and the state discriminator creates a competitive environment, with the VAE attempting to deceive the discriminator, while the state discriminator learns to differentiate between labeled and unlabeled inputs. Extensive evaluations on diverse datasets and ablation analysis affirm the effectiveness of the proposed model.

Figures (9)

• Figure 1: The proposed framework, named A Self-Supervised Framework for Learning Robust Representations for Active Learning (ADROIT), comprises a unified representation generator (VAE) with an encoder $(E)$ and a generator network $(G)$, a classifier network $(C)$, and a state discriminator $(D)$. The VAE learns a unified latent space using both labeled and unlabeled data. The classifier network incorporates annotation information into the latent space and refines it through self-supervised loss optimization with unlabeled data. Teacher-student training aligns the classifier $(C)$ with the target task-learner $(T)$. Meanwhile, the state discriminator distinguishes between labeled and unlabeled samples, aiding in the selection of informative unlabeled samples.
• Figure 2: Mean accuracy with standard deviations (shaded) of AL methods over the number of labeled samples on CIFAR10/100 (balanced), TinyImageNet-200 (balanced) and ImageNet-100 (balanced).
• Figure 3: Mean accuracy improvements with standard deviations (shaded) of AL methods w.r.t random sampling over the number of labeled samples on modified imbalanced CIFAR10 dataset.
• Figure 4: Mean accuracy improvements with standard deviations (shaded) of AL methods w.r.t random sampling over the number of labeled samples on Caltech101 dataset.
• Figure 5: Comparison of initialization methods on the performance of ADROIT using (balanced) CIFAR10 dataset.