Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ensembling Finetuned Language Models for Text Classification

Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka

TL;DR

A metadataset with predictions from five large finetuned models on six datasets is presented, and results of different ensembling strategies from these predictions are reported, shedding light on how ensembling can improve the performance of finetuned text classifiers and incentivize future adoption of ensembles in such tasks.

Abstract

Finetuning is a common practice widespread across different communities to adapt pretrained models to particular tasks. Text classification is one of these tasks for which many pretrained models are available. On the other hand, ensembles of neural networks are typically used to boost performance and provide reliable uncertainty estimates. However, ensembling pretrained models for text classification is not a well-studied avenue. In this paper, we present a metadataset with predictions from five large finetuned models on six datasets, and report results of different ensembling strategies from these predictions. Our results shed light on how ensembling can improve the performance of finetuned text classifiers and incentivize future adoption of ensembles in such tasks.

Ensembling Finetuned Language Models for Text Classification

TL;DR

A metadataset with predictions from five large finetuned models on six datasets is presented, and results of different ensembling strategies from these predictions are reported, shedding light on how ensembling can improve the performance of finetuned text classifiers and incentivize future adoption of ensembles in such tasks.

Abstract

Finetuning is a common practice widespread across different communities to adapt pretrained models to particular tasks. Text classification is one of these tasks for which many pretrained models are available. On the other hand, ensembles of neural networks are typically used to boost performance and provide reliable uncertainty estimates. However, ensembling pretrained models for text classification is not a well-studied avenue. In this paper, we present a metadataset with predictions from five large finetuned models on six datasets, and report results of different ensembling strategies from these predictions. Our results shed light on how ensembling can improve the performance of finetuned text classifiers and incentivize future adoption of ensembles in such tasks.

Paper Structure

This paper contains 14 sections, 1 equation, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Finetuning Text Classifiers (FTC) Metadataset
  4. Benchmarking Ensembles of Finetuned Text Classifiers
  5. Observation 1: Ensembling finetuned text classifiers is helpful.
  6. Observation 2: Ensembling text classifiers finetuned on 10% of the training data yields strong results.
  7. Conclusion
  8. Details on Datasets
  9. IMDB imdb_dataset
  10. Tweet tweet-sentiment-extraction
  11. DBpedia and News Zhang2015CharacterlevelCN
  12. SST-2 socher-etal-2013-recursive
  13. SetFit setfit-mnli
  14. Additional Results

Figures (4)

  • Figure 1: Mean error across datasets for different hyperparameter combinations.
  • Figure 2: Error for different hyperparameters using 10 % of the data.
  • Figure 3: Error vs. LoRA Rank, extended version. The error variation is small across different LoRA rank values.
  • Figure 4: Error vs. LoRA Rank, mini version. The error variation is small across different LoRA rank values.