Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Models for Text Classification: Is In-Context Learning Enough?

Aleksandra Edwards, Jose Camacho-Collados

TL;DR

The paper tackles the question of whether in-context learning with large language models can replace traditional fine-tuning for text classification in low-resource settings. It conducts a large-scale, cross-task comparison across 16 datasets and 7 domains, contrasting zero- and one-shot prompting of LLMs (LLaMA, Flan-T5, T5, GPT-3.5) with fine-tuned RoBERTa and FastText baselines. The findings show that while prompting with instruction-tuned models can outperform some baselines in low-resource scenarios, fine-tuned smaller models generally achieve higher accuracy, especially on multiclass and multilabel tasks, or when training data is available. The results emphasize that data efficiency and domain versatility of smaller, fine-tuned models currently surpass the out-of-the-box capabilities of prompting-based LLMs for text classification, though prompting remains a promising avenue in data-scarce contexts. These insights inform practical choices between prompt-based LLMs and traditional fine-tuning approaches in real-world, low-resource NLP applications.

Abstract

Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches based on fine-tuning is the ability to understand instructions written in natural language (prompts), which helps them generalise better to different tasks and domains without the need for specific training data. This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances. However, existing research is limited in scale and lacks understanding of how text generation models combined with prompting techniques compare to more established methods for text classification such as fine-tuning masked language models. In this paper, we address this research gap by performing a large-scale evaluation study for 16 text classification datasets covering binary, multiclass, and multilabel problems. In particular, we compare zero- and few-shot approaches of large language models to fine-tuning smaller language models. We also analyse the results by prompt, classification type, domain, and number of labels. In general, the results show how fine-tuning smaller and more efficient language models can still outperform few-shot approaches of larger language models, which have room for improvement when it comes to text classification.

Language Models for Text Classification: Is In-Context Learning Enough?

TL;DR

The paper tackles the question of whether in-context learning with large language models can replace traditional fine-tuning for text classification in low-resource settings. It conducts a large-scale, cross-task comparison across 16 datasets and 7 domains, contrasting zero- and one-shot prompting of LLMs (LLaMA, Flan-T5, T5, GPT-3.5) with fine-tuned RoBERTa and FastText baselines. The findings show that while prompting with instruction-tuned models can outperform some baselines in low-resource scenarios, fine-tuned smaller models generally achieve higher accuracy, especially on multiclass and multilabel tasks, or when training data is available. The results emphasize that data efficiency and domain versatility of smaller, fine-tuned models currently surpass the out-of-the-box capabilities of prompting-based LLMs for text classification, though prompting remains a promising avenue in data-scarce contexts. These insights inform practical choices between prompt-based LLMs and traditional fine-tuning approaches in real-world, low-resource NLP applications.

Abstract

Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches based on fine-tuning is the ability to understand instructions written in natural language (prompts), which helps them generalise better to different tasks and domains without the need for specific training data. This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances. However, existing research is limited in scale and lacks understanding of how text generation models combined with prompting techniques compare to more established methods for text classification such as fine-tuning masked language models. In this paper, we address this research gap by performing a large-scale evaluation study for 16 text classification datasets covering binary, multiclass, and multilabel problems. In particular, we compare zero- and few-shot approaches of large language models to fine-tuning smaller language models. We also analyse the results by prompt, classification type, domain, and number of labels. In general, the results show how fine-tuning smaller and more efficient language models can still outperform few-shot approaches of larger language models, which have room for improvement when it comes to text classification.
Paper Structure (22 sections, 5 figures, 6 tables)

This paper contains 22 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Classification
  4. Linear Methods
  5. Fine-tuning Methods
  6. Text Generation Models
  7. Prompting versus Fine-Tuning
  8. Experimental Setting
  9. Datasets
  10. Comparison Models
  11. Prompting, Training and Evaluation
  12. Prompt Design
  13. Results and Analysis
  14. Model and Prompt Analysis
  15. Prompting versus Fine-tuning
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Micro-F1 (left) and Macro-F1 (middle) results averaged across all datasets, comparing the performance of Flan-T5, LLaMA 1, and LLaMA 2 models for all three types of prompts, i.e., 'generic', 'task', and 'domain' as well as the average ('AVG') between them. 'Missing label' (right) shows the fraction of results returned by the three models that are different from the classification labels. Results are displayed for zero-shot ('zero') and one-shot setting ('one').
  • Figure 2: Comparison between prompting (left) and fine-tuning (right) approaches per text classification type where 'AVG' refers to averaged results across all prompt types per model. In 'Prompting', 'zero' and 'one' refer to zero- and one- shot prompt-based learning techniques, in 'Fine Tuning', 'one' refers to fine-tuning the models with one training instance per label and 'all' refers to fine-tuning using the entire dataset.
  • Figure 3: Wrong labels for prompting approaches per binary (left), multiclass (middle), and multilabel (right) classification where 'zero' refers to zero-shot learning and 'one' refers to one-shot learning.
  • Figure 4: Averaged Micro-F1 and Macro-F1 results based on number of classification labels: 'RoBERTa (all)' and 'T5 (all)' refer to models fine-tuned on the entire training set, 'Flan-T5 (one)' and 'LLaMA (one)' refer to one-shot prompting.
  • Figure 5: Comparison between prompting (left) and fine-tuning (right) approaches per text classification type where 'AVG' refers to averaged results across all prompt types per model. In 'Prompting', 'zero' and 'one' refer to zero- and one- shot prompt-based learning techniques, in 'Fine Tuning', 'one' refers to fine-tuning the models with one training instance per label and 'all' refers to fine-tuning using the entire dataset.