Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages

TL;DR

This work introduces Cendol, a collection of open instruction-tuned LLMs tailored for Indonesian languages, addressing representation gaps and safety concerns in low-resource contexts. It spans decoder-only and encoder-decoder architectures from 300M to 13B parameters and leverages a comprehensive Cendol Collection of ~53.5M prompts across NLP tasks, general knowledge, local languages, and human-centric prompts, implemented via a two-phase instruction-tuning regime. The results show about 20% improvements on NLU and NLG benchmarks and notable generalization to unseen tasks and indigenous languages, while highlighting the limited ability to capture local knowledge and cultural values and the inefficacy of parameter-efficient tuning like LoRA, with vocabulary adaptation proposed as an efficient alternative. Safety transferability from English pretraining to Indonesian is demonstrated, informing cross-language safety practices. Overall, Cendol provides a scalable path to higher-quality Indonesian LLMs, while identifying critical avenues for human alignment, local-knowledge capture, and culturally aware safety evaluation.

Abstract

Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder-decoder architectures across a range of model sizes. We highlight Cendol's effectiveness across a diverse array of tasks, attaining 20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning.

Figures (10)

• Figure 1: Overview of Cendol Collection and LLM adaptation into Cendol$^{inst}$ and Cendol$^{chat}$ models.
• Figure 2: The overview of Cendol Collection. Cendol Collection covers diverse prompts covering various types of instructions with a total of $\sim$53.5M prompts.
• Figure 3: Tasks and languages covered in our Indonesian indigenous benchmark evaluation suite.
• Figure 4: Performance comparison of Cendol models with various multilingual, Southeast Asian, and Indonesian LLMs on NLU tasks. Largest fully fine-tuned Cendol variants, i.e, Cendol mT5$_{XL}$ and Cendol LLaMA2 7B, significantly outperform existing LLMs by $\sim$20% weighted F1-score.
• Figure 5: Performance comparison of Cendol$^{inst}$ models with multilingual, SEA, and Indonesian LLMs on NLG tasks: (1) machine translation from local languages to Indonesian, (2) machine translation from Indonesian to local languages, (3) Indonesian language summarization, (4) Indonesian language question answering, and (5) Indonesian language paraphrasing. BLOOMZ and mT0 are not included since the evaluation datasets are exposed in xP3.