Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation

Nadezhda Chirkova, Vassilina Nikoulina, Jean-Luc Meunier, Alexandre Bérard

TL;DR

This work examines Sparse Mixture-of-Experts (SMoE) for multi-domain neural machine translation and compares it to straightforward width scaling of Transformers. It finds that width scaling often matches or exceeds SMoE performance while delivering far better inference efficiency on modern GPUs, suggesting SMoE offers limited practical gains in this setting. The study also evaluates domain knowledge integration strategies, finding that simple domain tags are nearly as effective as more complex gating schemes, and introduces domain randomization to improve robustness to unseen or mislabelled domains, which substantially helps without harming seen-domain performance. A key takeaway is that incorporating a large generic corpus (Paracrawl) and scaling the model are crucial for robust multi-domain translation, though the approach has limitations when adapting to entirely new domains not represented in training data.

Abstract

We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling, which helps to accommodate a variety of multi-domain data, and allow flexible sharing of parameters between domains, potentially enabling knowledge transfer between similar domains and limiting negative transfer. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE. We also search for a better recipe for robustness of multi-domain systems, highlighting the importance of mixing-in a generic domain, i.e. Paracrawl, and introducing a simple technique, domain randomization.

Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation

TL;DR

This work examines Sparse Mixture-of-Experts (SMoE) for multi-domain neural machine translation and compares it to straightforward width scaling of Transformers. It finds that width scaling often matches or exceeds SMoE performance while delivering far better inference efficiency on modern GPUs, suggesting SMoE offers limited practical gains in this setting. The study also evaluates domain knowledge integration strategies, finding that simple domain tags are nearly as effective as more complex gating schemes, and introduces domain randomization to improve robustness to unseen or mislabelled domains, which substantially helps without harming seen-domain performance. A key takeaway is that incorporating a large generic corpus (Paracrawl) and scaling the model are crucial for robust multi-domain translation, though the approach has limitations when adapting to entirely new domains not represented in training data.

Abstract

We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling, which helps to accommodate a variety of multi-domain data, and allow flexible sharing of parameters between domains, potentially enabling knowledge transfer between similar domains and limiting negative transfer. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE. We also search for a better recipe for robustness of multi-domain systems, highlighting the importance of mixing-in a generic domain, i.e. Paracrawl, and introducing a simple technique, domain randomization.
Paper Structure (33 sections, 2 equations, 4 figures, 20 tables)

This paper contains 33 sections, 2 equations, 4 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Multi-domain Neural Machine Translation.
  4. Mixture of Experts models for Neural Machine Translation.
  5. Methodology
  6. Mixture of Experts
  7. Domain Knowledge integration
  8. Domain Tags
  9. Domain Specialized Gate
  10. Domain-aware Gate.
  11. Domain randomization
  12. Experimental setup
  13. Data.
  14. Model hyperparameters.
  15. Models.
  16. ...and 18 more sections

Figures (4)

  • Figure 1:
  • Figure 2: Illustration of Mixture-of-Experts (left) and three considered ways of utilizing domain information in Mixture-of-Experts (right).
  • Figure 3: Pairwise similarity of experts activated with different domain embeddings.
  • Figure 4: Pairwise similarity of experts activated across different datasets, De-En translation. All the datasets were decoded with the generic domain embedding.