Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale

TL;DR

This paper explores an alternative approach to constructing a LLM for a new language by continually pre-training from existing pre-trained LLMs, instead of using randomly initialized parameters, and finds that CPT converges faster and saves significant resources in a scalable manner.

Abstract

In recent years, Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence. However, training these models from scratch requires substantial computational resources and vast amounts of text data. In this paper, we explore an alternative approach to constructing an LLM for a new language by continually pretraining (CPT) from existing pretrained LLMs, instead of using randomly initialized parameters. Based on parallel experiments on 40 model sizes ranging from 40M to 5B parameters, we find that 1) CPT converges faster and saves significant resources in a scalable manner; 2) CPT adheres to an extended scaling law derived from Hoffmann et al. (2022) with a joint data-parameter scaling term; 3) The compute-optimal data-parameter allocation for CPT markedly differs based on our estimated scaling factors; 4) The effectiveness of transfer at scale is influenced by training duration and linguistic properties, while robust to data replaying, a method that effectively mitigates catastrophic forgetting in CPT. We hope our findings provide deeper insights into the transferability of LLMs at scale for the research community.

Figures (6)

• Figure 1: Loss curves of pre-training and continual pre-training (CPT) across different model sizes. All models are pre-trained on Chinese text while CPT models are initialized from pre-trained English checkpoints. Dashed lines predict optimal loss at each computation level, as estimated in Section \ref{['sec:lc-re']}. (Left) Overlapped loss-compute power-law visualization, with each line representing one model. (Right) CPT LLM (2B parameters) reaches the same loss with approximately 50% fewer FLOPs.
• Figure 2: Reduced computational resources (top) and data consumption (bottom) with CPT. Only a subset of models of typical sizes is displayed for simplicity. (Top) Percentage reduction in FLOPs $C$ relative to pre-training from scratch $PT$, as estimated by $(C_{PT} - C_{CPT}) / C_{PT}$ at the same loss level for both strategies. (Bottom) Effectively Transferred Data, calculated by subtracting the tokens $D$ used by CPT from those used in pre-training from scratch at the same loss level, i.e. $D_{PT} - D_{CPT}$.
• Figure 3: Zero-shot evaluation for pre-trained and continually pre-trained (CPT) models of different languages. CPT models of various languages are initialized from the same checkpoint (light gray).
• Figure 4: Scaling of CPT with different English replaying ratios. Each blue line represents a 1.4B model continually pre-trained with various replaying ratios and evaluated on two validation sets: English (left) and Chinese (right). Models with English replaying ratios of 1%, 5%, 10%, 20%, 50%, and 80% are shown from light to dark blue, respectively. FLOPs allocated to each language are calculated by multiplying the corresponding language ratios by the total FLOPs.
• Figure 5: Predicted compute-optimal efficient frontiers on IsoLoss contour for both strategies.