OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Data

TL;DR

OpenSeal addresses the need for transparent, region-focused LLMs in Southeast Asia by studying how parallel data can extend an English-only LLM to multilingual SEA languages. The authors perform a controlled, large-scale continual pretraining study comparing five CPT data strategies, finding that parallel-only training yields the strongest cross-lingual transfer under a fixed token budget. They train OpenSeal, a 7B parameter model, on 34.7B tokens of parallel data at modest hardware cost (~$12k) and release fully open weights, data, and code. The results show competitive translation and multilingual reasoning compared with larger SEA LLMs built on multilingual bases, demonstrating a good, fast, and cheap path to high-quality, region-specific LLMs. The work highlights the value of parallel data for transparent, audit-friendly multilingual AI and sets a baseline for future instruction tuning and safety evaluation.

Abstract

Large language models (LLMs) have proven to be effective tools for a wide range of natural language processing (NLP) applications. Although many LLMs are multilingual, most remain English-centric and perform poorly on low-resource languages. Recently, several Southeast Asia-focused LLMs have been developed, but none are truly open source, as they do not publicly disclose their training data. Truly open-source models are important for transparency and for enabling a deeper and more precise understanding of LLM internals and development, including biases, generalization, and multilinguality. Motivated by recent advances demonstrating the effectiveness of parallel data in improving multilingual performance, we conduct controlled and comprehensive experiments to study the effectiveness of parallel data in continual pretraining of LLMs. Our findings show that using only parallel data is the most effective way to extend an LLM to new languages. Using just 34.7B tokens of parallel data and 180 hours on 8x NVIDIA H200 GPUs, we built OpenSeal, the first truly open Southeast Asian LLM that rivals the performance of existing models of similar size.

Figures (3)

• Figure 1: Illustration of the training data sequences of our parallel data investigation.
• Figure 2: Language probabilities of the Parallel Only (7B) model on the Chinese Cloze task across layers and training checkpoints.
• Figure 3: Language probabilities of the Multilingual (7B) model on the Chinese Cloze task across layers and training checkpoints.