Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment

Geyu Lin, Bin Wang, Zhengyuan Liu, Nancy F. Chen

TL;DR

The paper tackles the challenge of English-centric bias in multilingual LLMs caused by uneven data distribution during pre-training and instruction tuning. It introduces CrossIn, a cross-lingual instruction tuning framework that blends English-centric (Base) data with cross-lingual (CrossIn) and translation (Trans) data to align knowledge across languages, using three variants (en2x, x2en, x2x) and LoRA-based fine-tuning. A new Cross-Language Consistency Benchmark and Cross-XQuAD dataset are proposed to evaluate cross-lingual accuracy and consistency across reading comprehension, commonsense QA, and logic reasoning tasks, with AC3 as a combined metric. Experimental results across multiple base models and four languages show that CrossIn substantially improves cross-lingual consistency and task performance, with the x2x variant providing the strongest gains and translation data offering limited additional benefit, highlighting data-efficient multilingual alignment. These findings suggest practical routes to deploy more equitable multilingual LLMs and motivate further work on pretraining-stage effects and broader language coverage.

Abstract

Multilingual proficiency presents a significant challenge for large language models (LLMs). English-centric models are usually suboptimal in other languages, particularly those that are linguistically distant from English. This performance discrepancy mainly stems from the imbalanced distribution of training data across languages during pre-training and instruction tuning stages. To address this problem, we propose a novel approach called CrossIn, which utilizes a mixed composition of cross-lingual instruction tuning data. Our method leverages the compressed representation shared by various languages to efficiently enhance the model's task-solving capabilities and multilingual proficiency within a single process. In addition, we introduce a multi-task and multi-faceted benchmark to evaluate the effectiveness of CrossIn. Experimental results demonstrate that our method substantially improves performance across tasks and languages, and we provide extensive insights into the impact of cross-lingual data volume and the integration of translation data on enhancing multilingual consistency and accuracy.

CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment

TL;DR

The paper tackles the challenge of English-centric bias in multilingual LLMs caused by uneven data distribution during pre-training and instruction tuning. It introduces CrossIn, a cross-lingual instruction tuning framework that blends English-centric (Base) data with cross-lingual (CrossIn) and translation (Trans) data to align knowledge across languages, using three variants (en2x, x2en, x2x) and LoRA-based fine-tuning. A new Cross-Language Consistency Benchmark and Cross-XQuAD dataset are proposed to evaluate cross-lingual accuracy and consistency across reading comprehension, commonsense QA, and logic reasoning tasks, with AC3 as a combined metric. Experimental results across multiple base models and four languages show that CrossIn substantially improves cross-lingual consistency and task performance, with the x2x variant providing the strongest gains and translation data offering limited additional benefit, highlighting data-efficient multilingual alignment. These findings suggest practical routes to deploy more equitable multilingual LLMs and motivate further work on pretraining-stage effects and broader language coverage.

Abstract

Multilingual proficiency presents a significant challenge for large language models (LLMs). English-centric models are usually suboptimal in other languages, particularly those that are linguistically distant from English. This performance discrepancy mainly stems from the imbalanced distribution of training data across languages during pre-training and instruction tuning stages. To address this problem, we propose a novel approach called CrossIn, which utilizes a mixed composition of cross-lingual instruction tuning data. Our method leverages the compressed representation shared by various languages to efficiently enhance the model's task-solving capabilities and multilingual proficiency within a single process. In addition, we introduce a multi-task and multi-faceted benchmark to evaluate the effectiveness of CrossIn. Experimental results demonstrate that our method substantially improves performance across tasks and languages, and we provide extensive insights into the impact of cross-lingual data volume and the integration of translation data on enhancing multilingual consistency and accuracy.
Paper Structure (16 sections, 3 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 16 sections, 3 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Large Language Model
  4. Multilingual Evaluation Benchmark
  5. Cross-Lingual Consistency Benchmark
  6. Datasets and Metrics
  7. Cross-XQuAD Construction
  8. CrossIn Method
  9. Experiments
  10. Experimental Setting
  11. Main Results and Analysis
  12. Ablation Study
  13. Conclusion
  14. Appendix
  15. Prompt for Building Cross-XQuAD Data
  16. ...and 1 more sections

Figures (7)

  • Figure 1: An illustration of the dataset construction process of the Cross-XQuAD dataset. The original XQuAD dataset, although multilingual, is not adapted specifically to evaluate LLMs and their cross-lingual consistency.
  • Figure 2: Consistency score between languages on Cross-XQuAD with CrossIn$_{x2x}$ method
  • Figure 3: Results of different cross-lingual instruction tuning methods compared with baseline.
  • Figure 4: Comparison of AC3 score of adding translation data in cross-lingual instruction tuning.
  • Figure 5: Comparison of AC3 score by adding different numbers of CrossIn data. Base model: Mistral-7B-v0.1
  • ...and 2 more figures