The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classification

TL;DR

The paper extends a semi-supervised text classification framework (Cformer) by introducing CformerM, which pre-trains language models with objective masking derived from an unsupervised topic model (LDA). This masking targets topic-relevant words to improve topic sensitivity, and the teacher–student Meta Pseudo Labels framework is retained to leverage unlabeled data. Across English and Swedish datasets, CformerM generally outperforms Cformer and strong baselines, with larger gains when labeled data are scarce and in domain-specific tasks (e.g., Medical Abstracts). The work also analyzes the impact on reliability, interpretability, zero-shot performance, and practical training considerations, highlighting that effectiveness depends on dataset characteristics such as text length and topic coherence of the LDA-derived word lists.

Abstract

We extend and study a semi-supervised model for text classification proposed earlier by Hatefi et al. for classification tasks in which document classes are described by a small number of gold-labeled examples, while the majority of training examples is unlabeled. The model leverages the teacher-student architecture of Meta Pseudo Labels in which a ''teacher'' generates labels for originally unlabeled training data to train the ''student'' and updates its own model iteratively based on the performance of the student on the gold-labeled portion of the data. We extend the original model of Hatefi et al. by an unsupervised pre-training phase based on objective masking, and conduct in-depth performance evaluations of the original model, our extension, and various independent baselines. Experiments are performed using three different datasets in two different languages (English and Swedish).

Figures (5)

• Figure 1: A high-level overview of Random Masking in comparison to Objective Masking. While the former is used for general purpose language models, the latter is preferable for increasing the sensitivity of a language model to topical information.
• Figure 2: CformerM architecture: BERT encoders are pre-trained on the dataset via Objective Masking.
• Figure 3: The coherence diagram over a range of values of the number of topics for (a) Yahoo! Answers (b) AG News, (c) Bonnier News. Note that values on the $y$-axis range between $0.35$ and $0.6$. For each dataset, the range of values explored (resulting from the respective choice of $m$ and $k$) is denoted within parentheses in the legend. For instance, "Yahoo! Answers (5, 95, 5)" means that the number of topics ranges from 5 to 95 with a step size of 5, in this case, $m=5$ and $k=19$.
• Figure 4: Effect of batch-size per GPU on Cformer performance. A local batch size of 4 gives stable performance for the 10-sample case (left), while a batch size of 8 gives better performance for the 200-sample case (right). This suggests that when more training data is available, a larger batch size yields better performance.
• Figure 5: Effect of batch size and number of GPUs on CformerM performance. Similar to Cformer, the 10-sample case benefits from a smaller batch size, and CformerM will remain stable (and perform well) with 3 GPUs.