Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models

TL;DR

The paper tackles the challenge of extending math reasoning capabilities of large language models to non-English languages where task-specific data is scarce. It introduces a simple, post hoc layer-swapping method that combines two separately fine-tuned experts: a math expert trained on English math data and a language expert trained on generic target-language data, by reassembling the top and bottom layers from the language expert with the middle layers from the math expert. Empirical results on MGSM across Swahili, Telugu, Bengali, and Japanese show about a 10% performance gain over individual experts and model soup, with a transition zone added to smooth inter-layer interactions; the method also proves robust to interference and remains inexpensive. The work provides empirical evidence for cross-lingual patterns in the latent structures of LLMs and proposes a modular, post hoc approach to transferring reasoning capabilities across languages, potentially inspiring future language-specific adapters and modular architectures. Overall, layer swapping enables effective zero-shot cross-lingual transfer without target-language data and highlights interpretability opportunities in multilingual LLMs.

Abstract

Model merging, such as model souping, is the practice of combining different models with the same architecture together without further training. In this work, we present a model merging methodology that addresses the difficulty of fine-tuning Large Language Models (LLMs) for target tasks in non-English languages, where task-specific data is often unavailable. We focus on mathematical reasoning and without in-language math data, facilitate cross-lingual transfer by composing language and math capabilities. Starting from the same pretrained model, we fine-tune separate "experts" on math instruction data in English and on generic instruction data in the target language. We then replace the top and bottom transformer layers of the math expert directly with layers from the language expert, which consequently enhances math performance in the target language. The resulting merged models outperform the individual experts and other merging methods on the math benchmark, MGSM, by 10% across four major languages where math instruction data is scarce. In addition, this layer swapping is simple, inexpensive, and intuitive, as it is based on an interpretative analysis of the most important parameter changes during the fine-tuning of each expert. The ability to successfully re-compose LLMs for cross-lingual transfer in this manner opens up future possibilities to combine model expertise, create modular solutions, and transfer reasoning capabilities across languages all post hoc.

Figures (11)

• Figure 1: Our merging method which swaps in top and bottom transformer layers from a language expert into a math expert, buffered by a transition zone.
• Figure 2: Japanese expert #1
• Figure 3: Math expert #1
• Figure 5: Possible config. with more layers swapped in.
• Figure 6: Possible config. with less layers swapped in.