Training Deeper Neural Machine Translation Models with Transparent Attention

TL;DR

The paper tackles the difficulty of training very deep encoder stacks for neural machine translation. It introduces Transparent Attention, a lightweight mechanism that lets the decoder attend to a weighted mix of all encoder-layer outputs, improving gradient flow across depth. With this method, Transformer and RNMT+ encoders up to 20 layers train effectively and achieve BLEU gains of about 0.7–1.1 on WMT'14 En-De and WMT'15 Cs-En, often matching or surpassing Transformer Big with smaller capacity. The results indicate that deeper architectures can yield meaningful MT improvements when gradient pathways are properly regulated, motivating further exploration of very deep or wider models.

Abstract

While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.

Figures (4)

• Figure 1: Grad-norm ratio ($r_t$) vs training step ($t$) comparison for a 6 layer (blue) and 20 layer (red) Transformer trained on WMT 14 En$\rightarrow$De.
• Figure 2: Grad-norm ratio ($r_t$) vs training step ($t$) comparison for a 6 layer (blue) and 20 layer (red) RNMT+ model trained on WMT 14 En$\rightarrow$De.
• Figure 3: Grad-norm ratio ($r_t$) vs training step for 20 layer Transformer with transparent attention.
• Figure 4: Plot illustrating the variations in the learned attention weights $s_{i,6}$ for the 20 layer Transformer encoder over the training process.