Introducing Syllable Tokenization for Low-resource Languages: A Case Study with Swahili

TL;DR

This work introduces a syllable-based tokenizer for Swahili to address the limitations of generic subword tokenizers in low-resource, syllabic languages. It defines a 219-syllable Swahili syllable vocabulary and segmentation algorithm, then validates the approach by fine-tuning GPT-2 on Swahili data and comparing tokenizations (BPE, WordPiece, Syllable). Human evaluations show that syllable-based generation yields more fluent and structurally faithful Swahili text, supporting the claim that syllable embeddings can improve language modeling for such languages. The study demonstrates the practicality of syllable-aware tokenization for enhancing multilingual NLP with limited resources and motivates extending the approach to other African and syllable-rich languages.

Abstract

Many attempts have been made in multilingual NLP to ensure that pre-trained language models, such as mBERT or GPT2 get better and become applicable to low-resource languages. To achieve multilingualism for pre-trained language models (PLMs), we need techniques to create word embeddings that capture the linguistic characteristics of any language. Tokenization is one such technique because it allows for the words to be split based on characters or subwords, creating word embeddings that best represent the structure of the language. Creating such word embeddings is essential to applying PLMs to other languages where the model was not trained, enabling multilingual NLP. However, most PLMs use generic tokenization methods like BPE, wordpiece, or unigram which may not suit specific languages. We hypothesize that tokenization based on syllables within the input text, which we call syllable tokenization, should facilitate the development of syllable-aware language models. The syllable-aware language models make it possible to apply PLMs to languages that are rich in syllables, for instance, Swahili. Previous works introduced subword tokenization. Our work extends such efforts. Notably, we propose a syllable tokenizer and adopt an experiment-centric approach to validate the proposed tokenizer based on the Swahili language. We conducted text-generation experiments with GPT2 to evaluate the effectiveness of the syllable tokenizer. Our results show that the proposed syllable tokenizer generates syllable embeddings that effectively represent the Swahili language.

Figures (4)

• Figure 1: Example of a Swahili sentence (Sw) and the translation in English (En). The word "anakula" is broken down into syllables and explained in the bottom left table. We can replace the syllables, with syllables on the bottom right, creating new meanings. For example, the subject "a" can be replaced with "wa" resulting in "wanakula" to mean "They eat bread.". Similarly, the tense "na" can be replaced with "li" resulting in "alikula" to mean "She ate bread." This example demonstrates the richness of syllables in Swahili.
• Figure 2: Segmentation of Swahili words with a Syllable Tokenizer. There are five words in the input sentence. After segmentation, syllables indicated with the same color originate from the same word.
• Figure 3: Segmentation of Swahili words with a Syllable Tokenizer.
• Figure 4: Segmentation of Swahili words with a BERT Tokenizer.