Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TextGram: Towards a better domain-adaptive pretraining

Sharayu Hiwarkhedkar, Saloni Mittal, Vidula Magdum, Omkar Dhekane, Raviraj Joshi, Geetanjali Kale, Arnav Ladkat

TL;DR

This work tackles the environmental and computational costs of pretraining large transformer models by evaluating multiple data-selection strategies for domain-adaptive pretraining. It systematically compares N-grams, TF-IDF, perplexity, cross-entropy, TextRank, and Random Selection, and introduces TextGram, a TextRank-based method enhanced with in-domain n-gram cues and paraphrase mining to rank and select informative sentences. Empirically, data selection improves downstream text classification performance, with TextGram delivering the strongest results among the tested approaches while using a smaller, more domain-relevant pretraining corpus. The findings suggest that carefully selecting domain-relevant data can reduce training time and energy consumption without sacrificing accuracy, enabling more sustainable domain adaptation for NLP models.

Abstract

For green AI, it is crucial to measure and reduce the carbon footprint emitted during the training of large language models. In NLP, performing pre-training on Transformer models requires significant computational resources. This pre-training involves using a large amount of text data to gain prior knowledge for performing downstream tasks. Thus, it is important that we select the correct data in the form of domain-specific data from this vast corpus to achieve optimum results aligned with our domain-specific tasks. While training on large unsupervised data is expensive, it can be optimized by performing a data selection step before pretraining. Selecting important data reduces the space overhead and the substantial amount of time required to pre-train the model while maintaining constant accuracy. We investigate the existing selection strategies and propose our own domain-adaptive data selection method - TextGram - that effectively selects essential data from large corpora. We compare and evaluate the results of finetuned models for text classification task with and without data selection. We show that the proposed strategy works better compared to other selection methods.

TextGram: Towards a better domain-adaptive pretraining

TL;DR

This work tackles the environmental and computational costs of pretraining large transformer models by evaluating multiple data-selection strategies for domain-adaptive pretraining. It systematically compares N-grams, TF-IDF, perplexity, cross-entropy, TextRank, and Random Selection, and introduces TextGram, a TextRank-based method enhanced with in-domain n-gram cues and paraphrase mining to rank and select informative sentences. Empirically, data selection improves downstream text classification performance, with TextGram delivering the strongest results among the tested approaches while using a smaller, more domain-relevant pretraining corpus. The findings suggest that carefully selecting domain-relevant data can reduce training time and energy consumption without sacrificing accuracy, enabling more sustainable domain adaptation for NLP models.

Abstract

For green AI, it is crucial to measure and reduce the carbon footprint emitted during the training of large language models. In NLP, performing pre-training on Transformer models requires significant computational resources. This pre-training involves using a large amount of text data to gain prior knowledge for performing downstream tasks. Thus, it is important that we select the correct data in the form of domain-specific data from this vast corpus to achieve optimum results aligned with our domain-specific tasks. While training on large unsupervised data is expensive, it can be optimized by performing a data selection step before pretraining. Selecting important data reduces the space overhead and the substantial amount of time required to pre-train the model while maintaining constant accuracy. We investigate the existing selection strategies and propose our own domain-adaptive data selection method - TextGram - that effectively selects essential data from large corpora. We compare and evaluate the results of finetuned models for text classification task with and without data selection. We show that the proposed strategy works better compared to other selection methods.
Paper Structure (19 sections, 10 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 19 sections, 10 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Related Work
  4. Experimentation Setup
  5. Datasets
  6. Model Architecture
  7. Data Selection Techniques
  8. N-grams Coverage
  9. TF-IDF based selection
  10. Perplexity Based Data Selection
  11. Perplexity as the normalized inverse probability of the test set
  12. Cross Entropy
  13. TextRank
  14. Random Selection
  15. Proposed technique - TextGram
  16. ...and 4 more sections

Figures (2)

  • Figure 1: High-Level System Architecture Diagram - The corpus (both in-domain and out-domain) is first fed into pre-processing pipeline which will prepare the data for selection. Then, the selection strategy will be applied that will select data from out-domain based on in-domain training set. Further, selected corpus is used for continuous pre-training of BERT model. After pre-training, we perform fine-tuning to adapt the model on in-domain corpus.
  • Figure 2: Architecture diagram: TextGram based ranking