Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kydlíček, Zeyi Liu, Qunshu Lin, Sittipong Sripaisarnmongkol, Kridtaphad Sae-Khow, Nirattisai Thongchim, Taechawat Konkaew, Narong Borijindargoon, Anh Dao, Matichon Maneegard, Phakphum Artkaew, Zheng-Xin Yong, Quan Nguyen, Wannaphong Phatthiyaphaibun, Hoang H. Tran, Mike Zhang, Shiqi Chen, Tianyu Pang, Chao Du, Xinyi Wan, Wei Lu, Min Lin

TL;DR

Sailor2 addresses the underrepresentation of Southeast Asian languages in large language models by delivering an open family of multilingual LLMs (1B/8B/20B) trained with a data-centric pipeline built on Qwen2.5, including web and synthetic data, extensive cleaning, and a two-stage continual pre-training regime, followed by a two-stage post-training process (instruction and preference tuning) to deliver capable SEA-language performance. The authors also provide an open cookbook detailing data curation, model expansion, optimization, long-context training, speculative decoding, and pruning, along with comprehensive evaluations (SailCompass, SEA-WildBench, Flores Plus, and cultural benchmarks) showing Sailor2-20B achieving strong SEA-language results and competitive translations. Notable contributions include RegMix-based data mixture, anchor-based long-context training, and a language-consistency verifier to improve preference tuning, enabling robust multilingual capabilities for low-resource SEA languages and offering a practical blueprint for open multilingual LLM development. The work advances open-source, reproducible multilingual NLP in SEA, with implications for language development, cross-lingual translation, and cultural understanding, and provides guidelines for future improvements in synthetic data, tokenizer-free approaches, and efficient continual training.

Abstract

Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on Qwen2.5, Sailor2 undergoes continuous pre-training on 500B tokens (400B SEA-specific and 100B replay tokens) to support 13 SEA languages while retaining proficiency in Chinese and English. Sailor2-20B model achieves a 50-50 win rate against GPT-4o across SEA languages. We also deliver a comprehensive cookbook on how to develop the multilingual model in an efficient manner, including five key aspects: data curation, pre-training, post-training, model customization and evaluation. We hope that Sailor2 model (Apache 2.0 license) will drive language development in the SEA region, and Sailor2 cookbook will inspire researchers to build more inclusive LLMs for other under-served languages.

Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

TL;DR

Sailor2 addresses the underrepresentation of Southeast Asian languages in large language models by delivering an open family of multilingual LLMs (1B/8B/20B) trained with a data-centric pipeline built on Qwen2.5, including web and synthetic data, extensive cleaning, and a two-stage continual pre-training regime, followed by a two-stage post-training process (instruction and preference tuning) to deliver capable SEA-language performance. The authors also provide an open cookbook detailing data curation, model expansion, optimization, long-context training, speculative decoding, and pruning, along with comprehensive evaluations (SailCompass, SEA-WildBench, Flores Plus, and cultural benchmarks) showing Sailor2-20B achieving strong SEA-language results and competitive translations. Notable contributions include RegMix-based data mixture, anchor-based long-context training, and a language-consistency verifier to improve preference tuning, enabling robust multilingual capabilities for low-resource SEA languages and offering a practical blueprint for open multilingual LLM development. The work advances open-source, reproducible multilingual NLP in SEA, with implications for language development, cross-lingual translation, and cultural understanding, and provides guidelines for future improvements in synthetic data, tokenizer-free approaches, and efficient continual training.

Abstract

Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on Qwen2.5, Sailor2 undergoes continuous pre-training on 500B tokens (400B SEA-specific and 100B replay tokens) to support 13 SEA languages while retaining proficiency in Chinese and English. Sailor2-20B model achieves a 50-50 win rate against GPT-4o across SEA languages. We also deliver a comprehensive cookbook on how to develop the multilingual model in an efficient manner, including five key aspects: data curation, pre-training, post-training, model customization and evaluation. We hope that Sailor2 model (Apache 2.0 license) will drive language development in the SEA region, and Sailor2 cookbook will inspire researchers to build more inclusive LLMs for other under-served languages.

Paper Structure

This paper contains 67 sections, 2 equations, 16 figures, 29 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Open SEA Language Models
  4. Open SEA Language Resources
  5. Cookbook for Multilingual Language Models
  6. Data Curation
  7. Web Data Curation
  8. Synthetic Data Curation
  9. Data Cleaning
  10. Data Mixture
  11. Model Continual Pre-Training
  12. Model Expansion
  13. Model Parallel Optimization
  14. Zero Bubble Pipeline Parallelism
  15. Large Vocabulary Optimization
  16. ...and 52 more sections

Figures (16)

  • Figure 1: With rigorous data curation and efficient model expansion, Sailor2-20B achieves the 50-50 win rate over GPT4o on SEA languages, marking a new milestone of open LLMs.
  • Figure 2: Sailor2 Cookbook with key insights in data, model training and evaluation.
  • Figure 3: Distribution of categories and languages in SEA-UltraChat. Stage 2 data is carefully curated to ensure a balanced representation across both dimensions.
  • Figure 4: The PPL Percentile vs Reward Percentile distribution of English instruction data on Creative Tasks. We select High PPL High Reward candidates (top right) as stage 2 instruction data. We report corner cases highlighted in yellow in Table \ref{['tab:corner_case_analysis']}.
  • Figure 5: Comparison of GliDe Accept Length in Different Languages.
  • ...and 11 more figures