Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning

Ramakrishna Appicharla, Baban Gain, Santanu Pal, Asif Ekbal, Pushpak Bhattacharyya

TL;DR

This paper investigates context-aware neural machine translation (DocNMT) through cascade multi-task learning (MTL), treating context reconstruction as an auxiliary task alongside the main translation objective. By using a single encoder and two decoders, the model explicitly models context encoding via the reconstruction of the source from its context, with the objective $\mathcal{L} = \alpha \log p(y|x,c_x;\theta) + (1-\alpha) \log p(x|c_x;\theta)$ and $\alpha=0.5$. Experimental results on German→English across News, TED, and Europarl show that MTL improves translation quality in low-resource settings and that the context encoder tends to behave like a noise generator, similar to multi-encoder DocNMT, while failing to reconstruct the source from context, suggesting current document-level corpora may not be sufficiently context-aware. The study also finds that MTL is more sensitive to context choice than Inside-Context multi-encoder baselines, and pronoun translation accuracy (APT) generally benefits from the MTL approach in certain contexts. Overall, the work provides insights into the benefits and limitations of explicit context modeling via MTL in DocNMT, highlighting the potential and the need for dynamic training strategies and richer context data for robust context-aware translation.

Abstract

In document-level neural machine translation (DocNMT), multi-encoder approaches are common in encoding context and source sentences. Recent studies \cite{li-etal-2020-multi-encoder} have shown that the context encoder generates noise and makes the model robust to the choice of context. This paper further investigates this observation by explicitly modelling context encoding through multi-task learning (MTL) to make the model sensitive to the choice of context. We conduct experiments on cascade MTL architecture, which consists of one encoder and two decoders. Generation of the source from the context is considered an auxiliary task, and generation of the target from the source is the main task. We experimented with German--English language pairs on News, TED, and Europarl corpora. Evaluation results show that the proposed MTL approach performs better than concatenation-based and multi-encoder DocNMT models in low-resource settings and is sensitive to the choice of context. However, we observe that the MTL models are failing to generate the source from the context. These observations align with the previous studies, and this might suggest that the available document-level parallel corpora are not context-aware, and a robust sentence-level model can outperform the context-aware models.

A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning

TL;DR

This paper investigates context-aware neural machine translation (DocNMT) through cascade multi-task learning (MTL), treating context reconstruction as an auxiliary task alongside the main translation objective. By using a single encoder and two decoders, the model explicitly models context encoding via the reconstruction of the source from its context, with the objective and . Experimental results on German→English across News, TED, and Europarl show that MTL improves translation quality in low-resource settings and that the context encoder tends to behave like a noise generator, similar to multi-encoder DocNMT, while failing to reconstruct the source from context, suggesting current document-level corpora may not be sufficiently context-aware. The study also finds that MTL is more sensitive to context choice than Inside-Context multi-encoder baselines, and pronoun translation accuracy (APT) generally benefits from the MTL approach in certain contexts. Overall, the work provides insights into the benefits and limitations of explicit context modeling via MTL in DocNMT, highlighting the potential and the need for dynamic training strategies and richer context data for robust context-aware translation.

Abstract

In document-level neural machine translation (DocNMT), multi-encoder approaches are common in encoding context and source sentences. Recent studies \cite{li-etal-2020-multi-encoder} have shown that the context encoder generates noise and makes the model robust to the choice of context. This paper further investigates this observation by explicitly modelling context encoding through multi-task learning (MTL) to make the model sensitive to the choice of context. We conduct experiments on cascade MTL architecture, which consists of one encoder and two decoders. Generation of the source from the context is considered an auxiliary task, and generation of the target from the source is the main task. We experimented with German--English language pairs on News, TED, and Europarl corpora. Evaluation results show that the proposed MTL approach performs better than concatenation-based and multi-encoder DocNMT models in low-resource settings and is sensitive to the choice of context. However, we observe that the MTL models are failing to generate the source from the context. These observations align with the previous studies, and this might suggest that the available document-level parallel corpora are not context-aware, and a robust sentence-level model can outperform the context-aware models.
Paper Structure (24 sections, 6 equations, 3 figures, 9 tables)

This paper contains 24 sections, 6 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Statement
  5. Cascade Multi-Task Learning Transformer
  6. Context Selection
  7. Experiment Setup
  8. Vanilla-Sent:
  9. Concat-Context:
  10. Inside-Context:
  11. Data Statistics
  12. NMT Model Setups
  13. Results and Analysis
  14. Results of MTL and Contrastive Models
  15. Analysis of Reconstruction Objective
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The overview of our MTL architecture. The input to the model is a triplet. The triplet consist of (Context, Source, Target). The Intermediate Decoder is trained to reconstruct the Source given Context, and the Final Decoder is trained to translate the Source. Here, Source: Current source sentence, Context: Context for the current source sentence, and Target: Translation of current source sentence. None of the layers are shared.
  • Figure 2: The overview of the Inside-Context model. The input to the model is a triplet consisting of (Context, Source, Target). The multi-head attention layer of the decoder is modified to attend to both the context encoders ($\mathrm{Encoder_c}$) and the source encoder ($\mathrm{Encoder_s}$).
  • Figure 3: The overview of modified MTL architecture with residual connection. The input to the model is a triplet. The triplet consist of (Context, Source, Target) in Re-Src setting and (Source, Context, Target) in Re-Cntx setting. Here, Source: Current source sentence, Context: Context for the current source sentence, and Target: Translation of current source sentence. None of the layers are shared.