Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language

Sumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Ganesh Gautam, Usman Naseem

TL;DR

MaiBERT tackles the data scarcity of Maithili by building a large Maithili-specific corpus and pretraining a BERT-style model with Masked Language Modeling. The model, trained with a 30{,}000-token vocabulary and 512-token sequences, achieves 87.02% accuracy on Maithili news classification, surpassing strong baselines like NepBERTa and HindiBERT, and remains efficient with 0.11B parameters. The authors release maiBERT on Hugging Face, enabling downstream tasks such as sentiment analysis and NER, and demonstrate robust performance gains across classes. This work provides a practical, language-centered foundation for Maithili NLP and highlights the value of dedicated corpora and tokenization for low-resource languages.

Abstract

Natural Language Understanding (NLU) for low-resource languages remains a major challenge in NLP due to the scarcity of high-quality data and language-specific models. Maithili, despite being spoken by millions, lacks adequate computational resources, limiting its inclusion in digital and AI-driven applications. To address this gap, we introducemaiBERT, a BERT-based language model pre-trained specifically for Maithili using the Masked Language Modeling (MLM) technique. Our model is trained on a newly constructed Maithili corpus and evaluated through a news classification task. In our experiments, maiBERT achieved an accuracy of 87.02%, outperforming existing regional models like NepBERTa and HindiBERT, with a 0.13% overall accuracy gain and 5-7% improvement across various classes. We have open-sourced maiBERT on Hugging Face enabling further fine-tuning for downstream tasks such as sentiment analysis and Named Entity Recognition (NER).

MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language

TL;DR

MaiBERT tackles the data scarcity of Maithili by building a large Maithili-specific corpus and pretraining a BERT-style model with Masked Language Modeling. The model, trained with a 30{,}000-token vocabulary and 512-token sequences, achieves 87.02% accuracy on Maithili news classification, surpassing strong baselines like NepBERTa and HindiBERT, and remains efficient with 0.11B parameters. The authors release maiBERT on Hugging Face, enabling downstream tasks such as sentiment analysis and NER, and demonstrate robust performance gains across classes. This work provides a practical, language-centered foundation for Maithili NLP and highlights the value of dedicated corpora and tokenization for low-resource languages.

Abstract

Natural Language Understanding (NLU) for low-resource languages remains a major challenge in NLP due to the scarcity of high-quality data and language-specific models. Maithili, despite being spoken by millions, lacks adequate computational resources, limiting its inclusion in digital and AI-driven applications. To address this gap, we introducemaiBERT, a BERT-based language model pre-trained specifically for Maithili using the Masked Language Modeling (MLM) technique. Our model is trained on a newly constructed Maithili corpus and evaluated through a news classification task. In our experiments, maiBERT achieved an accuracy of 87.02%, outperforming existing regional models like NepBERTa and HindiBERT, with a 0.13% overall accuracy gain and 5-7% improvement across various classes. We have open-sourced maiBERT on Hugging Face enabling further fine-tuning for downstream tasks such as sentiment analysis and Named Entity Recognition (NER).

Paper Structure

This paper contains 14 sections, 4 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dataset
  4. Pre-training Corpus:
  5. Methodology
  6. Experimental Setup
  7. Evaluation Metrics
  8. Baselines
  9. Results and Discussion
  10. LLM Pre-training
  11. Downstream Classification and Comparison with SOTA
  12. Computational Experiments
  13. Traditional Machine Learning Baselines
  14. Conclusion and Future Works

Figures (6)

  • Figure 1: Density-Based Map of Maithili Language.
  • Figure 2: Comprehensive text statistics of the Maithili corpus showing character distribution, word counts, and linguistic features.
  • Figure 3: Word distribution patterns in Maithili corpus.
  • Figure 4: Distribution of news categories in the Maithili classification dataset.
  • Figure 5: Word count distribution across different news categories in the classification dataset.
  • ...and 1 more figures