Compass: Large Multilingual Language Model for South-east Asia

TL;DR

CompassLLM tackles the resource-scarce Southeast Asian language landscape by compiling a 1.7T-token multilingual pretraining corpus and applying curriculum-based strategies to emphasize low-resource languages. It combines supervised instruction fine-tuning with Direct Preference Optimization to align model outputs with human preferences, yielding strong performance, especially in Indonesian and Chinese contexts. Inference is made practical for deployment through context-length extension, accelerator techniques, and 4-bit weight quantization, enabling up to 128k context and efficient serving. The work demonstrates CompassLLM as a leading open-source foundation model for SEA languages, with significant commercial relevance for Shopee and broader SEA-language accessibility.

Abstract

Large language models have exhibited significant proficiency in languages endowed with extensive linguistic resources, such as English and Chinese. Nevertheless, their effectiveness notably diminishes when applied to languages characterized by limited linguistic resources, particularly within the Southeast Asian linguistic landscape, such as Indonesian. The scarcity of linguistic resources for these languages presents challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes. In response to these exigencies, we have introduced CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, with the primary aim of supporting the developmental requirements of Shopee. Our methodology encompasses several key strategies. To progressively enhance multilingual proficiencies, we implemented a multi-stage pre-training strategy integrated with curriculum learning, gradually intensifying the focus on low-resource languages. Concurrently, to better accommodate low-resource human instructions, we curated and generated a repository of high-quality multilingual human instructions, culminating the CompassLLM-SFT model through supervised instruction fine-tuning. Finally, to reinforce the model's alignment with human preference behaviors, we have embraced the principle of Direct Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary evaluation of the CompassLLM model yields promising results, with our model surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM, across diverse evaluation tasks, as verified through both automated and human-driven assessments. Notably, our model exhibits its superior performance in South-east Asia languages, such as Indonesian language.

Figures (16)

• Figure 1: Performance of CompassLLM, Falcon, LLaMA, SEA-LION, CompassLLM-SFT, CompassLLM-DPO, Falcon-Insturct, LLaMA-2-Chat, Vicuna-v1.3, Vicuna-v1.5, SEA-LION-Instruct, SeaLLM-chat. The solid lines represent pre-trained foundational LLMs, while the dashed lines represent LLMs optimized with alignment techniques. A lower "bias" score indicates better model performance, while higher scores for the other metrics suggest better performance. Experimental results show that our CompassLLM stands as the best foundational large language model on Southeast Asian, and has achieved better performance with other globally recognized open source LLMs.
• Figure 2: The overall construction process of CompassLLM. It mainly includes three stages. First, unsupervised learning is performed on a large corpus in the pre-training stage. Second, high-quality instruction data is created to fine-tune the pre-trained model with supervision. Finally, DPO learning is performed using human preference data.
• Figure 3: The overall pre-training data processing pipeline. The pipeline systematically processes the raw corpus data through a series of essential steps, including heuristic-based quality filtering, precise and fuzzy-match deduplication, language identification, data contamination mitigation, tokenizer training, and the application of language sampling methodologies.
• Figure 4: Composition of the pretraining dataset across source and language. (a) shows the source proportion, while (b) shows proportion of each language.
• Figure 5: To evaluate encoding compression rates of various tokenizers, we conducted experiments with 1 million documents from three languages - English, Chinese, and Indonesian. Our results indicate that CompassLLM, which focuses on efficient decoding for English, also maintains high compression ratios for both Chinese and Indonesian.