Effective Parallel Corpus Mining using Bilingual Sentence Embeddings

TL;DR

The paper introduces a scalable approach to parallel corpus mining using bilingual sentence embeddings learned with a dual-encoder architecture. By training with hard negatives and employing efficient ANN-based matching, it achieves strong sentence- and document-level mining, enabling high-quality mined data to train MT systems that approach the performance obtained with original corpora. It demonstrates robust reconstruction of the UN Parallel Corpus and competitive BLEU results on WMT datasets, while also offering an effective filtering method for ParaCrawl. The method delivers significant practical impact by reducing computational overhead and enhancing data quality for NMT pipelines, with strong alignment to human judgments in filtering tasks.

Abstract

This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings. Our embedding models are trained to produce similar representations exclusively for bilingual sentence pairs that are translations of each other. This is achieved using a novel training method that introduces hard negatives consisting of sentences that are not translations but that have some degree of semantic similarity. The quality of the resulting embeddings are evaluated on parallel corpus reconstruction and by assessing machine translation systems trained on gold vs. mined sentence pairs. We find that the sentence embeddings can be used to reconstruct the United Nations Parallel Corpus at the sentence level with a precision of 48.9% for en-fr and 54.9% for en-es. When adapted to document level matching, we achieve a parallel document matching accuracy that is comparable to the significantly more computationally intensive approach of [Jakob 2010]. Using reconstructed parallel data, we are able to train NMT models that perform nearly as well as models trained on the original data (within 1-2 BLEU).

Figures (4)

• Figure 1: Dual-encoder architecture, where a group of hidden layers encodes source sentence $x$ to $\mathbf{u}$ and a separate group encodes target sentence $y$ to $\mathbf{v}$ such that the score $\phi(x,y)$ is the dot-product ${\mathbf{u}}^T \cdot \mathbf{v}$.
• Figure 2: Matrix multiply trick for dot product model with random sampling.
• Figure 3: Scoring model based on dual-encoder architecture.
• Figure 4: Target sentence retrieval pipeline.