Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?

Pinzhen Chen, Simon Yu, Zhicheng Guo, Barry Haddow

TL;DR

This work investigates issues using controlled native or translated data during the instruction tuning and evaluation stages and shows that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot.

Abstract

Multilingual large language models are designed, claimed, and expected to cater to speakers of varied languages. We hypothesise that the current practices of fine-tuning and evaluating these models may not perfectly align with this objective owing to a heavy reliance on translation, which cannot cover language-specific knowledge but can introduce translation defects. It remains unknown whether the nature of the instruction data has an impact on the model output; conversely, it is questionable whether translated test sets can capture such nuances. Due to the often coupled practices of using translated data in both stages, such imperfections could have been overlooked. This work investigates these issues using controlled native or translated data during the instruction tuning and evaluation stages. We show that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot. The comparison between round-trip and single-pass translations reflects the importance of knowledge from language-native resources. Finally, we demonstrate that regularization is beneficial to bridging this gap on structured but not generative tasks.

Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?

TL;DR

This work investigates issues using controlled native or translated data during the instruction tuning and evaluation stages and shows that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot.

Abstract

Multilingual large language models are designed, claimed, and expected to cater to speakers of varied languages. We hypothesise that the current practices of fine-tuning and evaluating these models may not perfectly align with this objective owing to a heavy reliance on translation, which cannot cover language-specific knowledge but can introduce translation defects. It remains unknown whether the nature of the instruction data has an impact on the model output; conversely, it is questionable whether translated test sets can capture such nuances. Due to the often coupled practices of using translated data in both stages, such imperfections could have been overlooked. This work investigates these issues using controlled native or translated data during the instruction tuning and evaluation stages. We show that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot. The comparison between round-trip and single-pass translations reflects the importance of knowledge from language-native resources. Finally, we demonstrate that regularization is beneficial to bridging this gap on structured but not generative tasks.
Paper Structure (34 sections, 7 figures, 23 tables)

This paper contains 34 sections, 7 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction tuning data
  4. Multilingual LLM evaluation
  5. Experiment Design
  6. Instruction data
  7. Languages
  8. Native data
  9. Translated data
  10. Close-ended evaluation
  11. Native benchmarks
  12. Translated benchmarks
  13. Open-ended generation
  14. Experiments and Analysis
  15. Technical setup
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Results on native close-ended test sets: native instruction-tuned models have an edge.
  • Figure 2: Results on translated close-ended test sets: native instruction-tuned models are superior on XQuAD, but all data conditions have comparable results on MGSM and MT-MMLU.
  • Figure 3: Results on native and translated open-ended question answering: native instruction-tuned models are superior for translated questions when judged by GPT-4-Turbo, but all data conditions result in similar numbers in other cases.
  • Figure 4: Pearson's correlation between native data performance and native-translated performance difference for various benchmarks: weaker correlation for structured tasks and stronger correlation for generative tasks.
  • Figure 5: Round-trip translation (via English) produces translated data sharing the same origin as native data.
  • ...and 2 more figures