Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization

TL;DR

This work addresses the challenge of Arabic diacritization in text lacking diacritics by analyzing naturally occurring partial diacritics across six genres and introducing WildDiacs. It builds an extended hybrid analyze-and-disambiguate pipeline that leverages partial diacritics through Levenshtein-based re-ranking, morphological analyzer enhancements, and contextual post edits, validated on newly created maximally diacritized datasets Wild2MaxDiacs and WikiNewsMaxDiacs. The study provides a thorough data-centric view (datasets and statistics) and demonstrates significant gains in diacritization accuracy, highlighting the practical potential of wild diacritics to improve downstream NLP tasks. The open-source release and extensive dataset creation facilitate broader adoption and further improvements in Arabic diacritization and related applications such as TTS and ASR.

Abstract

The widespread absence of diacritical marks in Arabic text poses a significant challenge for Arabic natural language processing (NLP). This paper explores instances of naturally occurring diacritics, referred to as "diacritics in the wild," to unveil patterns and latent information across six diverse genres: news articles, novels, children's books, poetry, political documents, and ChatGPT outputs. We present a new annotated dataset that maps real-world partially diacritized words to their maximal full diacritization in context. Additionally, we propose extensions to the analyze-and-disambiguate approach in Arabic NLP to leverage these diacritics, resulting in notable improvements. Our contributions encompass a thorough analysis, valuable datasets, and an extended diacritization algorithm. We release our code and datasets as open source.

Figures (3)

• Figure 1: (a) The nine Arabic diacritics commonly used in Modern Standard Arabic, grouped by function; and four examples of diacritic clusters. (b) A visually annotated example of a diacritized phrase meaning 'and the bright suns [lit. and-the-suns the-bright]'. Diacritics are marked in red; and so are the undiacritized vowel-lengthening helping letters. Silent letters appear in dotted boxes.
• Figure 2: An example in different levels of diacritization. The red underlined diacritics highlight changes from row to row. ATB is the Arabic Treebank diacritization standard Maamouri:2004:penn, an essential full diacritization. WikiNews Darwish:2017:arabic-diac addresses some ATB gaps like missing Sukuns (1,2) and long vowel marking with aA (3). Maximal diacritization adds contextual diacritics in word-initial (4) and inter-word contexts (5).
• Figure 3: The original morphological analysis and disambiguation process with modifications and extensions.