Exploring Active Learning in Meta-Learning: Enhancing Context Set Labeling
Wonho Bae, Jing Wang, Danica J. Sutherland
TL;DR
This work investigates actively selecting labels for the context set in meta-learning. It demonstrates that active context selection at meta-training yields little to no gains, while deployment-time selection can significantly improve performance. The authors introduce a Gaussian Mixture Model–based selection method that leverages meta-learning representations, with a theoretical motivation showing Bayes-optimality under stylized conditions. Empirically, GMM-based active selection outperforms uncertainty-based and other low-budget strategies across few-shot classification, cross-domain tasks, and regression, across multiple meta-learning algorithms. The results offer a simple, robust, and broadly applicable approach to reducing labeling costs in real-world meta-learning systems.
Abstract
Most meta-learning methods assume that the (very small) context set used to establish a new task at test time is passively provided. In some settings, however, it is feasible to actively select which points to label; the potential gain from a careful choice is substantial, but the setting requires major differences from typical active learning setups. We clarify the ways in which active meta-learning can be used to label a context set, depending on which parts of the meta-learning process use active learning. Within this framework, we propose a natural algorithm based on fitting Gaussian mixtures for selecting which points to label; though simple, the algorithm also has theoretical motivation. The proposed algorithm outperforms state-of-the-art active learning methods when used with various meta-learning algorithms across several benchmark datasets.