A Survey of Text Classification Under Class Distribution Shift

TL;DR

This work surveys text classification under distribution shift, focusing on three core paradigms: learning with a background Universum class, zero-shot text classification, and open-set text classification with discovery. It provides a taxonomy of problem formulations, surveys mitigation strategies (e.g., OOD detection, energy-based losses, entailment-based zero-shot, and open-set pipelines), and highlights continual learning as a unifying direction to handle evolving class distributions. The study emphasizes the potential of continual learning to integrate the strengths of the three paradigms while mitigating their weaknesses, and discusses practical considerations, datasets, and limitations. Overall, the paper aims to guide future research towards open-set learning and discovery in NLP, with practical implications for robust, evolving text classification systems.

Abstract

The basic underlying assumption of machine learning (ML) models is that the training and test data are sampled from the same distribution. However, in daily practice, this assumption is often broken, i.e.~the distribution of the test data changes over time, which hinders the application of conventional ML models. One domain where the distribution shift naturally occurs is text classification, since people always find new topics to discuss. To this end, we survey research articles studying open-set text classification and related tasks. We divide the methods in this area based on the constraints that define the kind of distribution shift and the corresponding problem formulation, i.e.~learning with the Universum, zero-shot learning, and open-set learning. We next discuss the predominant mitigation approaches for each problem setup. Finally, we identify several future work directions, aiming to push the boundaries beyond the state of the art. Interestingly, we find that continual learning can solve many of the issues caused by the shifting class distribution. We maintain a list of relevant papers at https://github.com/Eduard6421/Open-Set-Survey.

Figures (3)

• Figure 1: An illustration of text classification by topic under class distribution shift, resulting in four class categories and associated learning problems. Known Known Classes (KKCs) correspond to standard supervised learning, Known Unknown Classes (KUCs) to learning with background class, Unknown Known Classes (UKCs) to zero-shot learning, and Unknown Unknown Classes (UUCs) to open-set learning. Continual learning provides viable solutions to mitigate class distribution shift. Best viewed in color.
• Figure 2: Zero-shot classification variants, as described by Yin-EMNLP-2019. In label-partially-unseen, the model is trained on labeled data from a subset of classes, then tested on the full set. In label-fully-unseen, the model is trained on unlabeled data, possibly unrelated to the target task. Best viewed in color.
• Figure 3: A generic open-set classification and discovery pipeline. The data is initially classified by the open-set classifier. The open-space instances are mined for new class candidates. Then, a zero-shot classifier is used to find the relevant instances. The process can be reiterated via continual learning, including the newly found classes and corresponding documents to retrain the original open-set model. Best viewed in color.