Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

Yaoyiran Li, Fangyu Liu, Ivan Vulić, Anna Korhonen

TL;DR

This work proposes a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking), applicable to any precalculated CLWE space, which improves their BLI capability and substantially outperforms a series of strong baselines across the board.

Abstract

Bilingual lexicon induction (BLI) with limited bilingual supervision is a crucial yet challenging task in multilingual NLP. Current state-of-the-art BLI methods rely on the induction of cross-lingual word embeddings (CLWEs) to capture cross-lingual word similarities; such CLWEs are obtained 1) via traditional static models (e.g., VecMap), or 2) by extracting type-level CLWEs from multilingual pretrained language models (mPLMs), or 3) through combining the former two options. In this work, we propose a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking), applicable to any precalculated CLWE space, which improves their BLI capability. The key idea is to 'extract' cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs. This crucial step is done via 1) creating a word similarity dataset, comprising positive word pairs (i.e., true translations) and hard negative pairs induced from the original CLWE space, and then 2) fine-tuning an mPLM (e.g., mBERT or XLM-R) in a cross-encoder manner to predict the similarity scores. At inference, we 3) combine the similarity score from the original CLWE space with the score from the BLI-tuned cross-encoder. BLICEr establishes new state-of-the-art results on two standard BLI benchmarks spanning a wide spectrum of diverse languages: it substantially outperforms a series of strong baselines across the board. We also validate the robustness of BLICEr with different CLWEs.

Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

TL;DR

This work proposes a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking), applicable to any precalculated CLWE space, which improves their BLI capability and substantially outperforms a series of strong baselines across the board.

Abstract

Bilingual lexicon induction (BLI) with limited bilingual supervision is a crucial yet challenging task in multilingual NLP. Current state-of-the-art BLI methods rely on the induction of cross-lingual word embeddings (CLWEs) to capture cross-lingual word similarities; such CLWEs are obtained 1) via traditional static models (e.g., VecMap), or 2) by extracting type-level CLWEs from multilingual pretrained language models (mPLMs), or 3) through combining the former two options. In this work, we propose a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking), applicable to any precalculated CLWE space, which improves their BLI capability. The key idea is to 'extract' cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs. This crucial step is done via 1) creating a word similarity dataset, comprising positive word pairs (i.e., true translations) and hard negative pairs induced from the original CLWE space, and then 2) fine-tuning an mPLM (e.g., mBERT or XLM-R) in a cross-encoder manner to predict the similarity scores. At inference, we 3) combine the similarity score from the original CLWE space with the score from the BLI-tuned cross-encoder. BLICEr establishes new state-of-the-art results on two standard BLI benchmarks spanning a wide spectrum of diverse languages: it substantially outperforms a series of strong baselines across the board. We also validate the robustness of BLICEr with different CLWEs.
Paper Structure (17 sections, 5 equations, 4 figures, 10 tables)

This paper contains 17 sections, 5 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction and Motivation
  2. Related Work
  3. Methodology
  4. Background
  5. blicer: Cross-Encoder Reranking
  6. Experimental Setup
  7. BLI Setups and Datasets
  8. CLWE Models (Baselines)
  9. blicer: Training Setup and Hyper-parameters
  10. Results and Discussion
  11. Conclusion
  12. Languages in Experiments
  13. Text Templates
  14. Unsupervised and Zero-shot Setups
  15. BLI Results on PanLex-BLI for Individual Language Pairs
  16. ...and 2 more sections

Figures (4)

  • Figure 1: An overview of the proposed blicer approach, described in detail in §\ref{['s:methodology']}.
  • Figure 2: Linear polarisation functions with different values of the hyper-parameter $\alpha$.
  • Figure 3: Average BLI scores on XLING with different values of the polarisation hyper-parameter $\alpha$.
  • Figure 4: Average BLI scores on XLING with different values of the interpolation hyper-parameter $\lambda$.