Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Probabilistic Models in Text Classification via Active Learning

Mitchell Bosley, Saki Kuzushima, Ted Enamorado, Yuki Shiraito

TL;DR

The paper tackles the high labeling cost of text classification in political science by introducing activeText, a semi-supervised mixture-model augmented with active learning. It combines labeled and unlabeled data through EM estimation and uncertainty-based document selection, with an optional keyword upweighting mechanism to inject domain knowledge. Empirical results show that activeText matches or exceeds the performance of Active SVM and can outperform BERT at low labeling budgets while dramatically reducing computation; the authors also demonstrate substantial replication efficiency in two published studies. The work contributes a practical, open-source tool that lowers the barrier to high-quality text classification in social science research and offers guidelines for tuning and robustness in real-world labeling tasks.

Abstract

Social scientists often classify text documents to use the resulting labels as an outcome or a predictor in empirical research. Automated text classification has become a standard tool, since it requires less human coding. However, scholars still need many human-labeled documents to train automated classifiers. To reduce labeling costs, we propose a new algorithm for text classification that combines a probabilistic model with active learning. The probabilistic model uses both labeled and unlabeled data, and active learning concentrates labeling efforts on difficult documents to classify. Our validation study shows that the classification performance of our algorithm is comparable to state-of-the-art methods at a fraction of the computational cost. Moreover, we replicate two recently published articles and reach the same substantive conclusions with only a small proportion of the original labeled data used in those studies. We provide activeText, an open-source software to implement our method.

Improving Probabilistic Models in Text Classification via Active Learning

TL;DR

The paper tackles the high labeling cost of text classification in political science by introducing activeText, a semi-supervised mixture-model augmented with active learning. It combines labeled and unlabeled data through EM estimation and uncertainty-based document selection, with an optional keyword upweighting mechanism to inject domain knowledge. Empirical results show that activeText matches or exceeds the performance of Active SVM and can outperform BERT at low labeling budgets while dramatically reducing computation; the authors also demonstrate substantial replication efficiency in two published studies. The work contributes a practical, open-source tool that lowers the barrier to high-quality text classification in social science research and offers guidelines for tuning and robustness in real-world labeling tasks.

Abstract

Social scientists often classify text documents to use the resulting labels as an outcome or a predictor in empirical research. Automated text classification has become a standard tool, since it requires less human coding. However, scholars still need many human-labeled documents to train automated classifiers. To reduce labeling costs, we propose a new algorithm for text classification that combines a probabilistic model with active learning. The probabilistic model uses both labeled and unlabeled data, and active learning concentrates labeling efforts on difficult documents to classify. Our validation study shows that the classification performance of our algorithm is comparable to state-of-the-art methods at a fraction of the computational cost. Moreover, we replicate two recently published articles and reach the same substantive conclusions with only a small proportion of the original labeled data used in those studies. We provide activeText, an open-source software to implement our method.
Paper Structure (26 sections, 4 equations, 6 figures, 2 tables, 1 algorithm)

This paper contains 26 sections, 4 equations, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Using Machine Learning for Text Classification
  3. Encoding Text in Matrix Form
  4. Supervised vs. Unsupervised Learning
  5. Discriminative vs. Generative Models
  6. Model Evaluation
  7. Active vs. Passive Learning
  8. The Method
  9. Model
  10. Active Learning
  11. Estimation
  12. Selection
  13. Labeling
  14. Stopping Rule
  15. Active Keyword Upweighting
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Passive vs Active Learning. For a classifier defined in two dimensions, Panel A illustrates the task: classify unlabeled documents (denoted by $\circ$ and $\ast$) as Political (P) or Non-political (N). A a passive learning algorithm will request the labels of $\circ$ and $\ast$ with equal probability (Panel C). In contrast, in active learning approach, $\circ$ will be prioritized for labeling as it is located in the region where the classifier is most uncertain (shaded region).
  • Figure 2: Comparison of Classification Results across Random and Active Versions of activeText and SVM
  • Figure 3: Comparison of Classification and Time Results across activeText, Active SVM, and BERT
  • Figure 4: Classification Results of activeText with and without Keywords
  • Figure 5: Replication of Figure 3 in gohdes2020repression: Expected Proportion of Target Killings, Given Internet Accessibility and Whether a Region is Inhabitated by the Alawi Minority. The results from activeText are presented in the left panel and those of gohdes2020repression are on the right.
  • ...and 1 more figures