Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Training for Multilingual Neural Machine Translation

Xinyi Wang, Yulia Tsvetkov, Graham Neubig

TL;DR

The paper tackles imbalanced data in multilingual MT by learning a language scorer through differentiable data selection, formulating a bi-level optimization that jointly updates the MT model and the language-sampling policy. It extends the Differentiable Data Selection framework to Multilingual DDS (MultiDDS), introducing a stabilized reward variant (MultiDDS-S) to improve multi-language optimization stability. Empirical results on 58-language TED data show that MultiDDS-S consistently outperforms heuristic baselines across one-to-many and many-to-one settings and provides controllable priorities for targeted language performance. The approach is model-agnostic, memory-efficient, and broadly applicable to multilingual tasks beyond MT, enabling robust, configurable cross-language transfer and performance balancing.

Abstract

When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others. Standard practice is to up-sample less resourced languages to increase representation, and the degree of up-sampling has a large effect on the overall performance. In this paper, we propose a method that instead automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages. Experiments on two sets of languages under both one-to-many and many-to-one MT settings show our method not only consistently outperforms heuristic baselines in terms of average performance, but also offers flexible control over the performance of which languages are optimized.

Balancing Training for Multilingual Neural Machine Translation

TL;DR

The paper tackles imbalanced data in multilingual MT by learning a language scorer through differentiable data selection, formulating a bi-level optimization that jointly updates the MT model and the language-sampling policy. It extends the Differentiable Data Selection framework to Multilingual DDS (MultiDDS), introducing a stabilized reward variant (MultiDDS-S) to improve multi-language optimization stability. Empirical results on 58-language TED data show that MultiDDS-S consistently outperforms heuristic baselines across one-to-many and many-to-one settings and provides controllable priorities for targeted language performance. The approach is model-agnostic, memory-efficient, and broadly applicable to multilingual tasks beyond MT, enabling robust, configurable cross-language transfer and performance balancing.

Abstract

When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others. Standard practice is to up-sample less resourced languages to increase representation, and the degree of up-sampling has a large effect on the overall performance. In this paper, we propose a method that instead automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages. Experiments on two sets of languages under both one-to-many and many-to-one MT settings show our method not only consistently outperforms heuristic baselines in terms of average performance, but also offers flexible control over the performance of which languages are optimized.

Paper Structure

This paper contains 28 sections, 11 equations, 5 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Multilingual Training Preliminaries
  3. Monolingual Training Objective
  4. Multilingual Training Formulation
  5. Evaluation Methods
  6. Relation to Heuristic Strategies
  7. Differentiable Data Selection
  8. DDS for Multilingual Training
  9. Method
  10. Implementing the Scorer Update
  11. Stabilized Multi-objective Training
  12. Experimental Evaluation
  13. Data and Settings
  14. Experiment Setup
  15. Baselines
  16. ...and 13 more sections

Figures (5)

  • Figure 1: An illustration of the MultiDDS algorithm. Solid lines represent updates for $\theta$, and dashed lines represent updates for $\psi$. The scorer defines the distribution over $n$ training languages, from which training data is sampled to train the model. The scorer is updated to favor the datasets with similar gradients as the gradient of the aggregated dev sets.
  • Figure 2: The difference between Low and High optimization objectives compared to Regular for the Diverse language group. MultiDDS successfully optimize for different priorities. left: M2O; right: O2M.
  • Figure 3: Language usage by training step. Left: many-to-one; Right: one-to-many; Top: related language group; Bottom: diverse language group.
  • Figure 4: Variance of reward. Left: M2O; Right: O2M; Top: Related language group; Bottom: Diverse language group.
  • Figure 5: Language usage for the M2O-Diverse setting. Left: MultiDDS-S; Right: MultiDDS. The two figures follow similar trends while MultiDDS changes more drastically.