Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

Lingyu Gao

TL;DR

This thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models by developing models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy.

Abstract

Text classification is crucial for applications such as sentiment analysis and toxic text filtering, but it still faces challenges due to the complexity and ambiguity of natural language. Recent advancements in deep learning, particularly transformer architectures and large-scale pretraining, have achieved inspiring success in NLP fields. Building on these advancements, this thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models (PLMs). Firstly, to address the challenge of selecting misleading yet incorrect distractors for cloze questions, we develop models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy. Secondly, to enhance model generalization to unseen labels, we create small finetuning datasets with domain-independent task label descriptions, improving model performance and robustness. Lastly, we tackle the sensitivity of large language models to in-context learning prompts by selecting effective demonstrations, focusing on misclassified examples and resolving model ambiguity regarding test example labels.

Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

TL;DR

This thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models by developing models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy.

Abstract

Text classification is crucial for applications such as sentiment analysis and toxic text filtering, but it still faces challenges due to the complexity and ambiguity of natural language. Recent advancements in deep learning, particularly transformer architectures and large-scale pretraining, have achieved inspiring success in NLP fields. Building on these advancements, this thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models (PLMs). Firstly, to address the challenge of selecting misleading yet incorrect distractors for cloze questions, we develop models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy. Secondly, to enhance model generalization to unseen labels, we create small finetuning datasets with domain-independent task label descriptions, improving model performance and robustness. Lastly, we tackle the sensitivity of large language models to in-context learning prompts by selecting effective demonstrations, focusing on misclassified examples and resolving model ambiguity regarding test example labels.
Paper Structure (124 sections, 7 equations, 14 figures, 49 tables)

This paper contains 124 sections, 7 equations, 14 figures, 49 tables.

Table of Contents

  1. Introduction
  2. Feature engineering with PLMs for distractor selection
  3. Improving robustness for prompt-based zero-shot classification
  4. Demonstration selection for in-context learning
  5. Organization of the Thesis
  6. Text Classification with Pretrained Language Models
  7. Pretrained Language Models
  8. ELMo
  9. GPT and BERT
  10. GPT.
  11. BERT.
  12. Comparison to ELMo.
  13. Other Recent PLMs
  14. Text Classification with PLMs
  15. Finetuning
  16. ...and 109 more sections

Figures (14)

  • Figure 1: Label-specific feature histograms for MCDSent (inflected form pairs).
  • Figure 2: Label-specific feature histograms for MCDSent (headword pairs).
  • Figure 3: Label-specific feature histograms for MCDPara (inflected form pairs).
  • Figure 4: Label-specific feature histograms for MCDPara (headword pairs).
  • Figure 5: Histograms of BERT log-probabilities of selected distractors ("T"), unselected distractors ("F"), and correct answers ("$c$") in MCDSent.
  • ...and 9 more figures