Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset

Simon P. Ramalepe, Thipe I. Modipa, Marelie H. Davel

TL;DR

This paper tackles data scarcity in Sepedi by assembling two corpora, SepMono and SepNews, and training GPT-2–style generative models. It systematically compares standard (non-occlusion) pre-training with occlusion-based pre-training, followed by fine-tuning on SepNews using gradual unfreezing. The study reports that non-occlusion pre-training achieves lower validation loss and perplexity, while occlusion pre-training yields higher BLEU on generated text, with fine-tuning narrowing performance gaps and producing strong BLEU scores (44.98% for SepGPT and 48.84% for SepGPT-OCC). Overall, the work provides new Sepedi baselines and demonstrates that occlusion-based training can enhance text generation quality on limited data, offering practical pathways for low-resource language NLP.

Abstract

Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.

Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset

TL;DR

This paper tackles data scarcity in Sepedi by assembling two corpora, SepMono and SepNews, and training GPT-2–style generative models. It systematically compares standard (non-occlusion) pre-training with occlusion-based pre-training, followed by fine-tuning on SepNews using gradual unfreezing. The study reports that non-occlusion pre-training achieves lower validation loss and perplexity, while occlusion pre-training yields higher BLEU on generated text, with fine-tuning narrowing performance gaps and producing strong BLEU scores (44.98% for SepGPT and 48.84% for SepGPT-OCC). Overall, the work provides new Sepedi baselines and demonstrates that occlusion-based training can enhance text generation quality on limited data, offering practical pathways for low-resource language NLP.

Abstract

Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.
Paper Structure (19 sections, 4 equations, 2 figures, 5 tables)

This paper contains 19 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Pre-trained language models
  4. Closely related studies
  5. Experimental setup
  6. Data collection
  7. SepMono Dataset
  8. SepNews dataset
  9. Data cleaning and pre-processing
  10. Models
  11. Model architecture
  12. Model training
  13. Occlusion-based training
  14. Fine-tuning models
  15. Evaluation metrics
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Comparison of training curves: (a) Occlusion-based and (b) Non-occlusion.
  • Figure 2: Occlusion hyperparameter search