ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs

TL;DR

ELO tackles the computational bottlenecks and degradation issues in continual pretraining of multilingual LLMs by detaching and training only the first and last layers for a target language, then reintegrating them through a concise alignment step. The method comprises ELO Pretraining on detached layers and Layer Alignment to fuse the learned knowledge back into the full model, augmented by bilingual instruction tuning. Empirical results show up to 6.46x training speedups versus FFT and up to 6.2% qualitative gains on target languages, while largely preserving English capabilities; ablations confirm the importance of layer choice, alignment data, and data scale. This approach provides a practical, data-efficient path for improving targeted language performance in multilingual models without prohibitive compute costs.

Abstract

We propose an efficient layer-specific optimization (ELO) method designed to enhance continual pretraining (CP) for specific languages in multilingual large language models (MLLMs). This approach addresses the common challenges of high computational cost and degradation of source language performance associated with traditional CP. The ELO method consists of two main stages: (1) ELO Pretraining, where a small subset of specific layers, identified in our experiments as the critically important first and last layers, are detached from the original MLLM and trained with the target language. This significantly reduces not only the number of trainable parameters but also the total parameters computed during the forward pass, minimizing GPU memory consumption and accelerating the training process. (2) Layer Alignment, where the newly trained layers are reintegrated into the original model, followed by a brief full fine-tuning step on a small dataset to align the parameters. Experimental results demonstrate that the ELO method achieves a training speedup of up to 6.46 times compared to existing methods, while improving target language performance by up to 6.2\% on qualitative benchmarks and effectively preserving source language (English) capabilities.

Figures (3)

• Figure 1: Description of the proposed ELO training process
• Figure 2: Performance variation of LogicKor based on the amount of PT data for its layer aligning
• Figure 3: Comparison of the training time across ELO, FFT, and LoRA training methods