Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CATT: Character-based Arabic Tashkeel Transformer

Faris Alasmary, Orjuwan Zaafarani, Ahmad Ghannam

TL;DR

This work tackles Arabic Text Diacritization (ATD) by pretraining a character-based BERT and fine-tuning encoder-only and encoder-decoder transformers for ATD, followed by Noisy-Student self-training. The proposed approach achieves state-of-the-art performance on two benchmarks, WikiNews and the newly introduced CATT, with relative improvements in $DER$ of 30.83% on WikiNews and 35.21% on CATT, and even surpasses $GPT$-4-turbo on CATT by 9.36% in $DER$. Key findings show that MLM pretraining, longer training, and NS each contribute substantial gains, and the combination of these techniques yields the strongest results. The authors also release their CATT benchmark and models to foster reproducibility and future research in Arabic ATD, with implications for downstream tasks like text-to-speech and machine translation.

Abstract

Tashkeel, or Arabic Text Diacritization (ATD), greatly enhances the comprehension of Arabic text by removing ambiguity and minimizing the risk of misinterpretations caused by its absence. It plays a crucial role in improving Arabic text processing, particularly in applications such as text-to-speech and machine translation. This paper introduces a new approach to training ATD models. First, we finetuned two transformers, encoder-only and encoder-decoder, that were initialized from a pretrained character-based BERT. Then, we applied the Noisy-Student approach to boost the performance of the best model. We evaluated our models alongside 11 commercial and open-source models using two manually labeled benchmark datasets: WikiNews and our CATT dataset. Our findings show that our top model surpasses all evaluated models by relative Diacritic Error Rates (DERs) of 30.83\% and 35.21\% on WikiNews and CATT, respectively, achieving state-of-the-art in ATD. In addition, we show that our model outperforms GPT-4-turbo on CATT dataset by a relative DER of 9.36\%. We open-source our CATT models and benchmark dataset for the research community\footnote{https://github.com/abjadai/catt}.

CATT: Character-based Arabic Tashkeel Transformer

TL;DR

This work tackles Arabic Text Diacritization (ATD) by pretraining a character-based BERT and fine-tuning encoder-only and encoder-decoder transformers for ATD, followed by Noisy-Student self-training. The proposed approach achieves state-of-the-art performance on two benchmarks, WikiNews and the newly introduced CATT, with relative improvements in of 30.83% on WikiNews and 35.21% on CATT, and even surpasses -4-turbo on CATT by 9.36% in . Key findings show that MLM pretraining, longer training, and NS each contribute substantial gains, and the combination of these techniques yields the strongest results. The authors also release their CATT benchmark and models to foster reproducibility and future research in Arabic ATD, with implications for downstream tasks like text-to-speech and machine translation.

Abstract

Tashkeel, or Arabic Text Diacritization (ATD), greatly enhances the comprehension of Arabic text by removing ambiguity and minimizing the risk of misinterpretations caused by its absence. It plays a crucial role in improving Arabic text processing, particularly in applications such as text-to-speech and machine translation. This paper introduces a new approach to training ATD models. First, we finetuned two transformers, encoder-only and encoder-decoder, that were initialized from a pretrained character-based BERT. Then, we applied the Noisy-Student approach to boost the performance of the best model. We evaluated our models alongside 11 commercial and open-source models using two manually labeled benchmark datasets: WikiNews and our CATT dataset. Our findings show that our top model surpasses all evaluated models by relative Diacritic Error Rates (DERs) of 30.83\% and 35.21\% on WikiNews and CATT, respectively, achieving state-of-the-art in ATD. In addition, we show that our model outperforms GPT-4-turbo on CATT dataset by a relative DER of 9.36\%. We open-source our CATT models and benchmark dataset for the research community\footnote{https://github.com/abjadai/catt}.
Paper Structure (11 sections, 3 equations, 2 figures, 11 tables)

This paper contains 11 sections, 3 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Preparation
  4. Training Data
  5. Benchmark Data
  6. Experiments
  7. Results
  8. RQ1: Does the ATD model benefit from MLM pretraining?
  9. RQ2: Does training ATD model for more iterations help?
  10. RQ3: Is the NS approach effective in ATD models?
  11. Conclusion

Figures (2)

  • Figure 1: Encoder-Decoder (ED) Model
  • Figure 2: Encoder-Only (EO) Model