Enhancing Context Through Contrast

TL;DR

The paper addresses the challenge of achieving language-agnostic yet semantically rich representations for neural machine translation. It introduces Context Enhancement (CE), which leverages the BarLow Twins loss $\mathcal{L_{BT}}$ to maximize mutual information between parallel sentences treated as different views of the same meaning, without explicit data augmentation. By fine-tuning only the encoder with CE and then training a standard NMT decoder, the approach aims to improve translation while promoting language-agnostic embeddings derived from pre-trained models. Empirical findings show potential gains on MT benchmarks, but also reveal limitations due to dataset compatibility and the sensitivity of pre-trained embeddings, underscoring the need for careful selection of pre-trained features and data pipelines.

Abstract

Neural machine translation benefits from semantically rich representations. Considerable progress in learning such representations has been achieved by language modelling and mutual information maximization objectives using contrastive learning. The language-dependent nature of language modelling introduces a trade-off between the universality of the learned representations and the model's performance on the language modelling tasks. Although contrastive learning improves performance, its success cannot be attributed to mutual information alone. We propose a novel Context Enhancement step to improve performance on neural machine translation by maximizing mutual information using the Barlow Twins loss. Unlike other approaches, we do not explicitly augment the data but view languages as implicit augmentations, eradicating the risk of disrupting semantic information. Further, our method does not learn embeddings from scratch and can be generalised to any set of pre-trained embeddings. Finally, we evaluate the language-agnosticism of our embeddings through language classification and use them for neural machine translation to compare with state-of-the-art approaches.

Figures (1)

• Figure 1: A block diagram of our proposed architecture for the CE step. The encoder maps sentences ($\boldsymbol{\tilde{x}}, \boldsymbol{\tilde{y}}$) to sequences of latent representations ($\boldsymbol{\tilde{\omega}}^S, \boldsymbol{\tilde{\omega}}^T$). These are then aggregated to get sentence embeddings $\boldsymbol{\tilde{\sigma}}^S = \phi(\boldsymbol{\tilde{\omega}}^S)$ and $\boldsymbol{\tilde{\sigma}}^T = \phi(\boldsymbol{\tilde{\omega}}^T)$. The contrastive loss $\mathcal{L_{BT}}$, is applied to batch normalized projections $\mathbf{Z}^S = \mathtt{BN}(\rho(\boldsymbol{\sigma}^S; \theta_\rho))$ and $\mathbf{Z}^T = \mathtt{BN}(\rho(\boldsymbol{\sigma}^T; \theta_\rho))$. The weights $\theta_E$, are fine tuned for NMT after the CE step. Also, $\boldsymbol{\tilde{\omega}}^S$ is directly passed to the decoder while training on NMT.