Improving Word Translation via Two-Stage Contrastive Learning

TL;DR

This work proposes a robust and effective two-stage contrastive learning framework for the BLI task, and proposes to refine standard cross-lingual linear maps between static word embeddings (WEs) via a Contrastive learning objective.

Abstract

Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI task. At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps. In Stage C2, we conduct BLI-oriented contrastive fine-tuning of mBERT, unlocking its word translation capability. We also show that static WEs induced from the `C2-tuned' mBERT complement static WEs from Stage C1. Comprehensive experiments on standard BLI datasets for diverse languages and different experimental setups demonstrate substantial gains achieved by our framework. While the BLI method from Stage C1 already yields substantial gains over all state-of-the-art BLI methods in our comparison, even stronger improvements are met with the full two-stage framework: e.g., we report gains for 112/112 BLI setups, spanning 28 language pairs.

Figures (16)

• Figure 1: An illustration of the proposed two-stage BLI approach (see §\ref{['s:methodology']}). It combines contrastive tuning on both static WEs (C1) and pretrained multilingual LMs (C2), where the static WEs are leveraged for selecting negative examples in contrastive tuning of the LM. The output of C1 and C2 is combined for the final BLI task.
• Figure 2: BLI scores with different $\lambda$ values: (left) $|\mathcal{D}_0|$=$5k$; (middle) $|\mathcal{D}_0|$=$1k$; (right) PanLex-BLI, $|\mathcal{D}_0|$=$1k$.
• Figure 3: A t-SNE visualisation tsne:2012 of mBERT encodings of words from BLI test sets for ru-it (left) and tr-hr (right). Similar plots for more language pairs are in Appendix \ref{['appendix:tsne']}.
• Figure 4: A t-SNE visualisation tsne:2012 of mapped fastText WEs of words from BLI test sets for ru-it (left) and tr-hr (right). Similar plots for more language pairs are in Appendix \ref{['appendix:tsne_fastText']}.
• Figure 5: A t-SNE visualisation of mBERT-encoded representations of words from the EN-IT BLI test set. The representations before BLI-oriented fine-tuning of mBERT in Stage C2 are plotted in muted blue and red, and after fine-tuning in bright colours.