Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

Alexander Arno Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, Mehdi Ali

TL;DR

This work is the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages.

Abstract

The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.

Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

TL;DR

This work is the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages.

Abstract

The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.
Paper Structure (46 sections, 8 figures, 11 tables)

This paper contains 46 sections, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction-tuning
  4. Template-based Datasets
  5. Synthetic Datasets
  6. Multilingual Datasets
  7. Evaluation
  8. Pre-training Benchmarks
  9. Instruction-tuning Benchmarks
  10. Demarcation
  11. Multilingual Instruction-Tuning Data
  12. Bactrian-X
  13. Lima-X
  14. Language Mixtures
  15. Multilingual Benchmarks
  16. ...and 31 more sections

Figures (8)

  • Figure 1: Percentage improvement for turn one averaged across MT-Bench-X languages of models fine-tuned on parallel mixed language instruction-tuning datasets over single language fine-tunings.
  • Figure 2: Pair-wise MT-Bench-DE quality assessment by humans and GPT-4, including voting option "both bad".
  • Figure 3: Bactrian-DE vs. Bactrian-ENDEFRITES. Pair-wise MT-Bench-DE quality assessment by GPT-4.
  • Figure 4: GPT-4-as-a-judge single evaluation average scores for each language mix dataset variant on MT-Bench-X.
  • Figure 5: In-depth MT-Bench-X quality assessment by GPT-4.
  • ...and 3 more figures