A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correction

TL;DR

CAMEL introduces a confidence-based, pool-based active learning framework for sequential multi-output tasks that reduces expert labeling by combining selective human annotation with self-supervision, label validation, and efficient post-hoc uncertainty estimation. The four-stage loop—data selection, labeling, label validation, and semi-supervised retraining—enables strong data efficiency across tasks like machine translation and dialogue belief tracking. A label-correction component further improves dataset quality by detecting and replacing potentially erroneous human annotations. Across translation and dialogue tracking experiments, CAMEL demonstrates superior labeling efficiency and robust performance, with optional label validation (CAMELL) enhancing dataset quality via online corrections. The framework generalizes beyond these domains, leveraging calibrated uncertainty estimation to guide learning with limited supervision and noisy labels.

Abstract

Supervised neural approaches are hindered by their dependence on large, meticulously annotated datasets, a requirement that is particularly cumbersome for sequential tasks. The quality of annotations tends to deteriorate with the transition from expert-based to crowd-sourced labelling. To address these challenges, we present CAMEL (Confidence-based Acquisition Model for Efficient self-supervised active Learning), a pool-based active learning framework tailored to sequential multi-output problems. CAMEL possesses two core features: (1) it requires expert annotators to label only a fraction of a chosen sequence, and (2) it facilitates self-supervision for the remainder of the sequence. By deploying a label correction mechanism, CAMEL can also be utilised for data cleaning. We evaluate CAMEL on two sequential tasks, with a special emphasis on dialogue belief tracking, a task plagued by the constraints of limited and noisy datasets. Our experiments demonstrate that CAMEL significantly outperforms the baselines in terms of efficiency. Furthermore, the data corrections suggested by our method contribute to an overall improvement in the quality of the resulting datasets.

Figures (8)

• Figure 1: CAMEL comprises four stages. Stage 1 involves data selection, choosing instances for labelling where the model shows uncertainty (confidence below the $\alpha_{\text{sel}}$ threshold), as indicated by pink arrows. In Stage 2, annotators label the selected instances while the model self-labels the remaining ones (dashed green arrows). Stage 3 (optional) validates labels using a label confidence estimate, incorporating only labels exceeding the $\alpha_{\text{val}}$ threshold and the self-labelled data into the dataset (black arrows). Finally, Stage 4 involves retraining the model for the next cycle.
• Figure 2: Category-specific uncertainty measures: (a) displays prediction uncertainty, including prediction probability and total and knowledge uncertainty; (b) depicts label uncertainty, including label probability and total and knowledge uncertainty from both learning and noisy models.
• Figure 3: The model-based annotation process for semi-supervised annotation for NMT. The learning model initiates the translation with the word "The", then confidence for the next token generation is below the threshold. The expert annotation model is prompted and provides the next word, "drunks". The learning model resumes and successfully generates the remainder of the translation: "interrupted the event".
• Figure 4: METEOR score of the T5 translation model using different active learning approaches on the WMT$17$ DE-EN test set, as a function of (a) the number of word-level labels and (b) the number of complete translations, with $95\%$ confidence interval.
• Figure 5: COMET score of the T5 translation model using different active learning approaches on the WMT$17$ DE-EN test set, as a function of (a) the number of word-level labels and (b) the number of complete translations, with $95\%$ confidence interval.