Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Could We Have Had Better Multilingual LLMs If English Was Not the Central Language?

Ryandito Diandaru, Lucky Susanto, Zilu Tang, Ayu Purwarianti, Derry Wijaya

TL;DR

The paper investigates whether English is the optimal central language for multilingual LLMs by evaluating Llama2's translation across 41 languages and linking MT performance to URIEL-based linguistic distances. It demonstrates that increasing model scale, more than instruction tuning or few-shot prompts, yields the largest gains for unseen languages, while correlations between linguistic proximity and translation quality depend on multiple features beyond syntax. The findings show that some languages with less training data (e.g., Swedish, Catalan) exhibit correlation patterns similar to English, challenging the assumption that English must be the central hub for multilingual models. These results suggest that non-English-centered multilingual training could improve data efficiency and broaden accessibility for multilingual NLP applications.

Abstract

Large Language Models (LLMs) demonstrate strong machine translation capabilities on languages they are trained on. However, the impact of factors beyond training data size on translation performance remains a topic of debate, especially concerning languages not directly encountered during training. Our study delves into Llama2's translation capabilities. By modeling a linear relationship between linguistic feature distances and machine translation scores, we ask ourselves if there are potentially better central languages for LLMs other than English. Our experiments show that the 7B Llama2 model yields above 10 BLEU when translating into all languages it has seen, which rarely happens for languages it has not seen. Most translation improvements into unseen languages come from scaling up the model size rather than instruction tuning or increasing shot count. Furthermore, our correlation analysis reveals that syntactic similarity is not the only linguistic factor that strongly correlates with machine translation scores. Interestingly, we discovered that under specific circumstances, some languages (e.g. Swedish, Catalan), despite having significantly less training data, exhibit comparable correlation levels to English. These insights challenge the prevailing landscape of LLMs, suggesting that models centered around languages other than English could provide a more efficient foundation for multilingual applications.

Could We Have Had Better Multilingual LLMs If English Was Not the Central Language?

TL;DR

The paper investigates whether English is the optimal central language for multilingual LLMs by evaluating Llama2's translation across 41 languages and linking MT performance to URIEL-based linguistic distances. It demonstrates that increasing model scale, more than instruction tuning or few-shot prompts, yields the largest gains for unseen languages, while correlations between linguistic proximity and translation quality depend on multiple features beyond syntax. The findings show that some languages with less training data (e.g., Swedish, Catalan) exhibit correlation patterns similar to English, challenging the assumption that English must be the central hub for multilingual models. These results suggest that non-English-centered multilingual training could improve data efficiency and broaden accessibility for multilingual NLP applications.

Abstract

Large Language Models (LLMs) demonstrate strong machine translation capabilities on languages they are trained on. However, the impact of factors beyond training data size on translation performance remains a topic of debate, especially concerning languages not directly encountered during training. Our study delves into Llama2's translation capabilities. By modeling a linear relationship between linguistic feature distances and machine translation scores, we ask ourselves if there are potentially better central languages for LLMs other than English. Our experiments show that the 7B Llama2 model yields above 10 BLEU when translating into all languages it has seen, which rarely happens for languages it has not seen. Most translation improvements into unseen languages come from scaling up the model size rather than instruction tuning or increasing shot count. Furthermore, our correlation analysis reveals that syntactic similarity is not the only linguistic factor that strongly correlates with machine translation scores. Interestingly, we discovered that under specific circumstances, some languages (e.g. Swedish, Catalan), despite having significantly less training data, exhibit comparable correlation levels to English. These insights challenge the prevailing landscape of LLMs, suggesting that models centered around languages other than English could provide a more efficient foundation for multilingual applications.
Paper Structure (14 sections, 3 figures, 5 tables)

This paper contains 14 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Machine Translation Evaluation
  4. Correlation Score Analysis
  5. Results and Analysis
  6. Machine Translation Evaluation Results
  7. Scaling up the model enhances translation ability. However, improvements from instruction-tuning and adding shot count remain inconclusive.
  8. Language Importance Analysis
  9. Syntactic similarity may be an important feature, but other linguistic dimensions can be too.
  10. English is not always the most syntactically important.
  11. Related Work
  12. Conclusion
  13. Bibliographical References
  14. Language Resource References

Figures (3)

  • Figure 1: Scatter plot for inllama and outllama languages against the SYNTACTIC distance to Swedish. The correlation score is -0.67 and the p-value is 3.16e-06. The negative correlation here implies that the smaller the SYNTACTIC distance of a language to Swedish, the better is its MT performance
  • Figure 2: Heatmaps of correlations between linguistic distances with BLEU scores of the Llama2-7B one-shot prompting setup (language subset considered is written above each heatmap)
  • Figure 3: Heatmaps of correlations between linguistic distances with COMET-22 scores of the Llama2-7B one-shot prompting setup (language subset considered is written above each heatmap)