XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners

TL;DR

This work addresses the limitations of uncertainty-only active learning in low-resource text classification by introducing Explainable Active Learning (XAL). XAL jointly trains an encoder for classification and a decoder for generating explanations, with a ranking loss to align explanations with human reasoning, and selects unlabeled data using a blend of predictive uncertainty and explanation quality. Empirical results across six tasks show consistent gains over nine baselines, with strong data-efficiency and high interpretability as evidenced by human evaluation and visualization analyses. The approach demonstrates that leveraging rationales and external explanations can yield more informative data selection and better generalization in resource-constrained settings, albeit with extra computational costs for explanation generation.

Abstract

Active learning (AL), which aims to construct an effective training set by iteratively curating the most formative unlabeled data for annotation, has been widely used in low-resource tasks. Most active learning techniques in classification rely on the model's uncertainty or disagreement to choose unlabeled data, suffering from the problem of over-confidence in superficial patterns and a lack of exploration. Inspired by the cognitive processes in which humans deduce and predict through causal information, we take an initial attempt towards integrating rationales into AL and propose a novel Explainable Active Learning framework (XAL) for low-resource text classification, which aims to encourage classifiers to justify their inferences and delve into unlabeled data for which they cannot provide reasonable explanations. Specifically, besides using a pre-trained bi-directional encoder for classification, we employ a pre-trained uni-directional decoder to generate and score the explanation. We further facilitate the alignment of the model with human reasoning preference through a proposed ranking loss. During the selection of unlabeled data, the predicted uncertainty of the encoder and the explanation score of the decoder complement each other as the final metric to acquire informative data. Extensive experiments on six datasets show that XAL achieves consistent improvement over 9 strong baselines. Analysis indicates that the proposed method can generate corresponding explanations for its predictions.

Figures (10)

• Figure 1: Data selection strategy in AL. Previous work selects the unlabeled data mostly relying on the model's uncertainly (a), but we propose to further leverage the model's explanation of its prediction (b).
• Figure 2: Our proposed XAL framework, which can be divided into two main parts -- the training process (red arrows) and the data selection process (blue arrows). The training process aims to train the encoder-decoder model to learn classification and explanation generation. The data selection process aims to select unlabeled data using predictive entropy and explanation scores.
• Figure 3: Results given the data selection budget 500 instances in six text classification tasks, where 100 instances are selected for annotation in each iteration. Here we plot the specific values of XAL and the second significant performance when using 500 instances, and the detailed performance values can be found in Appendix \ref{['MainDetails']}.
• Figure 4: Experimental results demonstrate how much data, when selected using AL methods, is required for the models to achieve 90% of the performance of those trained on the complete training datasets. In each iteration, we annotate 50 instances. The performance of models trained on the whole training sets is, (a) RTE -- 83.11%, (b) MRPC -- 84.74%, (c) COVID19 -- 75.45%, and (d) DEBA -- 65.71%. The green triangles refer to the average values of the experiments on three different initial sets $D_l$ and three different random seeds. The circles refer to outliers. Detailed results can be seen in Appendix \ref{['UPPERBOUND']}.
• Figure 5: Results of ablation study in the six text classification tasks. We select 100 instances in each iteration and conduct 4 iterations (the same with Section \ref{['Budget']}). The results are measured using macro-F1 scores and they are the average values on three different initial sets $D_l$ and three different random seeds.