Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FaBERT: Pre-training BERT on Persian Blogs

Mostafa Masumi, Seyed Soroush Majd, Mehrnoush Shamsfard, Hamid Beigy

TL;DR

FaBERT addresses the need for compact, effective Persian NLU models capable of handling informal and formal text without relying on large LLMs. It adopts a BERT-base architecture with a 50k WordPiece vocabulary, pre-trained from the cleaned HmBlogs Persian blogs corpus using MLM with dynamic masking and without Next Sentence Prediction, totaling 400 hours on an Nvidia A100. Evaluated across 12 datasets spanning NLI, QP, NER, SA, and QA, FaBERT achieves improvements in several tasks and generally ranks among the top performers while maintaining a small parameter footprint of 124M. The results highlight the value of clean, diverse blog data for Persian NLP and demonstrate FaBERT’s practical applicability for efficient, high-quality Persian language understanding and downstream applications.

Abstract

We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs corpus, encompassing both informal and formal Persian texts. FaBERT is designed to excel in traditional Natural Language Understanding (NLU) tasks, addressing the intricacies of diverse sentence structures and linguistic styles prevalent in the Persian language. In our comprehensive evaluation of FaBERT on 12 datasets in various downstream tasks, encompassing Sentiment Analysis (SA), Named Entity Recognition (NER), Natural Language Inference (NLI), Question Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated improved performance, all achieved within a compact model size. The findings highlight the importance of utilizing diverse and cleaned corpora, such as HmBlogs, to enhance the performance of language models like BERT in Persian Natural Language Processing (NLP) applications. FaBERT is openly accessible at https://huggingface.co/sbunlp/fabert

FaBERT: Pre-training BERT on Persian Blogs

TL;DR

FaBERT addresses the need for compact, effective Persian NLU models capable of handling informal and formal text without relying on large LLMs. It adopts a BERT-base architecture with a 50k WordPiece vocabulary, pre-trained from the cleaned HmBlogs Persian blogs corpus using MLM with dynamic masking and without Next Sentence Prediction, totaling 400 hours on an Nvidia A100. Evaluated across 12 datasets spanning NLI, QP, NER, SA, and QA, FaBERT achieves improvements in several tasks and generally ranks among the top performers while maintaining a small parameter footprint of 124M. The results highlight the value of clean, diverse blog data for Persian NLP and demonstrate FaBERT’s practical applicability for efficient, high-quality Persian language understanding and downstream applications.

Abstract

We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs corpus, encompassing both informal and formal Persian texts. FaBERT is designed to excel in traditional Natural Language Understanding (NLU) tasks, addressing the intricacies of diverse sentence structures and linguistic styles prevalent in the Persian language. In our comprehensive evaluation of FaBERT on 12 datasets in various downstream tasks, encompassing Sentiment Analysis (SA), Named Entity Recognition (NER), Natural Language Inference (NLI), Question Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated improved performance, all achieved within a compact model size. The findings highlight the importance of utilizing diverse and cleaned corpora, such as HmBlogs, to enhance the performance of language models like BERT in Persian Natural Language Processing (NLP) applications. FaBERT is openly accessible at https://huggingface.co/sbunlp/fabert
Paper Structure (15 sections, 5 figures, 11 tables)

This paper contains 15 sections, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Training Corpus
  5. Pre-training Procedure
  6. Experiments and Results
  7. Natural Language Inference and Question Paraphrasing
  8. Named Entity Recognition
  9. Sentiment Analysis and Irony Detection
  10. Question Answering
  11. Vocabulary Impact on Input Length
  12. Conclusion
  13. Fine-tuning Hyperparameters
  14. Detailed NER Results
  15. Tokenizer Comparison Figures

Figures (5)

  • Figure 1: Train and Validation MLM loss in pre-training
  • Figure 2: Token count distribution across tokenizers for the PQuAD dataset
  • Figure 3: Token count distribution across model tokenizers for the ParsiNLU Reading Comprehension dataset
  • Figure 4: Token count distribution across tokenizers for the MirasOpinion dataset
  • Figure 5: Token count distribution across tokenizers for the FarsTail dataset