Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

TL;DR

The paper tackles zero-shot text classification by standardizing datasets and evaluations across topic, emotion, and situation, and by recasting the problem as textual entailment. It introduces Definition-Wild as a harder, real-world setting with no label-specific training data and demonstrates a unified entailment-based approach that converts labels into hypotheses and leverages BERT-based models trained on MNLI, FEVER, and RTE. The work provides three benchmark datasets with train/dev/test splits and seen/unseen partitions, analyzes two evaluation regimes (partially and fully unseen), and shows that entailment models can robustly handle unseen labels, especially when guided by effective hypothesis generation and ensembling. The public release of code and data supports reproducibility and future research in open-domain zero-shot classification.

Abstract

Zero-shot text classification (0Shot-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0Shot-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0Shot-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0Shot-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress. This work benchmarks the 0Shot-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0Shot-TC relative to conceptually different and diverse aspects: the ``topic'' aspect includes ``sports'' and ``politics'' as labels; the ``emotion'' aspect includes ``joy'' and ``anger''; the ``situation'' aspect includes ``medical assistance'' and ``water shortage''. ii) We extend the existing evaluation setup (label-partially-unseen) -- given a dataset, train on some labels, test on all labels -- to include a more challenging yet realistic evaluation label-fully-unseen 0Shot-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0Shot-TC of diverse aspects within a textual entailment formulation and study it this way. Code & Data: https://github.com/yinwenpeng/BenchmarkingZeroShot

Figures (2)

• Figure 1: A piece of text can be assigned labels which describe the different aspects of the text. Positive labels are in blue.
• Figure 2: Performance of different situation classes in label-fully-unseen, predicted by the ensemble model.