NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

TL;DR

NusaBERT addresses Indonesia's rich and diverse linguistic landscape by extending IndoBERT with vocabulary expansion and continued pre-training on a carefully curated multilingual corpus that emphasizes regional languages. It trains two model variants, NusaBERT_BASE and NusaBERT_LARGE, using a RoBERTa-style MLM objective and a 128-token sequence length, evaluating on IndoNLU, NusaX, and NusaWrites to assess NLU, multilinguality, and multiculturality. The approach yields competitive results across Indonesian tasks and state-of-the-art performance on many NusaX languages, with notable gains in sequence labeling and some translation-oriented tasks, though classification tasks show mixed results and code-switching remains challenging. Limitations include incomplete coverage of intra-sentential code-switching, domain diversity gaps, and reliance on existing corpora; future work suggests expanding languages via lexicons and adapters and incorporating more diverse data sources such as Bible translations or speech transcripts to better capture Indonesia's linguistic reality.

Abstract

Indonesia's linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world's most linguistically rich nations. This diversity, coupled with the widespread practice of code-switching and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages and dialects. Through rigorous evaluation across a range of benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks involving multiple languages of Indonesia, paving the way for future natural language understanding research for under-represented languages.