Active Learning with Weak Supervision for Gaussian Processes

TL;DR

The paper extends Gaussian Process active learning to a weakly supervised setting by enabling simultaneous selection of data points and annotation precision within a budget. It introduces a BALD-inspired acquisition that optimizes mutual information per annotation cost, modeling weak labels with variance terms that scale as $\gamma/\alpha$. Empirical results on synthetic data, UCI benchmarks, and classification tasks demonstrate that acquiring many low-precision annotations can outperform a few high-precision ones by enabling broader input-space exploration, with gains particularly pronounced when annotation costs are uneven. The approach is general to regression and classification, leveraging EP for GP classification to obtain tractable MI estimates, and offers a practical framework for cost-aware active learning in real-world annotation pipelines.

Abstract

Annotating data for supervised learning can be costly. When the annotation budget is limited, active learning can be used to select and annotate those observations that are likely to give the most gain in model performance. We propose an active learning algorithm that, in addition to selecting which observation to annotate, selects the precision of the annotation that is acquired. Assuming that annotations with low precision are cheaper to obtain, this allows the model to explore a larger part of the input space, with the same annotation budget. We build our acquisition function on the previously proposed BALD objective for Gaussian Processes, and empirically demonstrate the gains of being able to adjust the annotation precision in the active learning loop.

Figures (7)

• Figure 1: Generative models for the weak target variable, $\widetilde{Y}$. (a) Generative model without $Y$. (b) $\widetilde{Y}$ is conditionally independent of $f$ and $X$ given $Y$. (c) $\widetilde{Y}$ and $Y$ are independent given $f$ and $X$.
• Figure 2: Median, first and third quartiles of the test MSE obtained from each set of experiments. Left: Sine curve experiments using cost functions with varying parameter $q$. Middle: Sine curve experiments with an under-explored input space and where $\widetilde{Y}$ is independent of $f$ and $X$ given $Y$. Right: UCI data experiments.
• Figure 3: Top: Examples of each of the three artificial classification data sets. Negative labels are given in blue and positive in red. Bottom: Median, first and third quartiles of the test accuracy obtained from each set of experiments.
• Figure 4: Left and middle: Median, first and third quartiles of the test MSE obtained from each set of 15 experiments on the sine curve data set, for different values of the parameter $q$ of the cost function. Right: Annotation precisions selected by $\text{MI}(\widetilde{Y}; f)$ and $\text{MI}(\widetilde{Y}; Y)$ for $q=0.8$. Shown are the selected precision levels over the course of two experiments (Experiment 1 and Experiment 2, respectively).
• Figure 5: Effect of learning the hyperparameters of the GP kernel in the active learning experiments performed on the sine curve data set. Shown is median, first and third quartiles of the test MSE obtained from each set of 15 experiments.