Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation

Jonas Belouadi, Steffen Eger

TL;DR

This work develops fully unsupervised evaluation metrics that leverage similarities and synergies between evaluation metric induction, parallel corpus mining, and MT systems and induce unsuper supervised multilingual sentence embeddings from pseudo-parallel data.

Abstract

The vast majority of evaluation metrics for machine translation are supervised, i.e., (i) are trained on human scores, (ii) assume the existence of reference translations, or (iii) leverage parallel data. This hinders their applicability to cases where such supervision signals are not available. In this work, we develop fully unsupervised evaluation metrics. To do so, we leverage similarities and synergies between evaluation metric induction, parallel corpus mining, and MT systems. In particular, we use an unsupervised evaluation metric to mine pseudo-parallel data, which we use to remap deficient underlying vector spaces (in an iterative manner) and to induce an unsupervised MT system, which then provides pseudo-references as an additional component in the metric. Finally, we also induce unsupervised multilingual sentence embeddings from pseudo-parallel data. We show that our fully unsupervised metrics are effective, i.e., they beat supervised competitors on 4 out of our 5 evaluation datasets. We make our code publicly available.

USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation

TL;DR

This work develops fully unsupervised evaluation metrics that leverage similarities and synergies between evaluation metric induction, parallel corpus mining, and MT systems and induce unsuper supervised multilingual sentence embeddings from pseudo-parallel data.

Abstract

The vast majority of evaluation metrics for machine translation are supervised, i.e., (i) are trained on human scores, (ii) assume the existence of reference translations, or (iii) leverage parallel data. This hinders their applicability to cases where such supervision signals are not available. In this work, we develop fully unsupervised evaluation metrics. To do so, we leverage similarities and synergies between evaluation metric induction, parallel corpus mining, and MT systems. In particular, we use an unsupervised evaluation metric to mine pseudo-parallel data, which we use to remap deficient underlying vector spaces (in an iterative manner) and to induce an unsupervised MT system, which then provides pseudo-references as an additional component in the metric. Finally, we also induce unsupervised multilingual sentence embeddings from pseudo-parallel data. We show that our fully unsupervised metrics are effective, i.e., they beat supervised competitors on 4 out of our 5 evaluation datasets. We make our code publicly available.
Paper Structure (35 sections, 6 equations, 5 figures, 8 tables)

This paper contains 35 sections, 6 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. XMoverScore ()
  4. WMD
  5. Vector space remapping
  6. Language Model
  7. DistilScore
  8. SentSim ()
  9. Methods
  10. [wrd]
  11. Efficient WMD Pseudo-Parallel Data Mining
  12. Pseudo References
  13. [snt]
  14. Contrastive Learning
  15. Ratio Margin Pseudo-Parallel Data Mining
  16. ...and 20 more sections

Figures (5)

  • Figure 1: Relationship between metrics $\operatorname{m}$, vector spaces, parallel data, and MT systems: Metrics build on (potentially deficient) multilingual vector spaces (a), and can be used to mine (pseudo-)parallel sentences (b), which in turn can be used to improve deficient vector spaces (c). (Pseudo-)parallel data can also be used to train MT systems (d), which can generate pseudo-references (e). Conversely, metrics can also be optimization criteria for MT systems, which in turn can generate additional pseudo-parallel data through translation (f & g; not explored in this work).
  • Figure 2: [wrd] with pseudo references (left), unsupervised remapping (middle) and a LM (right).
  • Figure 3: Results for unsupervised vector space remapping (top) and contrastive learning with [snt] (bottom) for de-en. Pearson's r is computed on WMT-16 (MT evaluation) and P@1 on News Commentary v15 (parallel sentence matching).
  • Figure 4: Influence of a language model and an MT system on [wrd], segment-level Pearson's r for different values of $w_{\text{pseudo}}$ (weight for pseudo references) and $w_{\text{lm}}$ (weight for the language model) on WMT-16 de-en data. Note that the point $w_{\text{pseudo}}=w_{\text{lm}}=0$ uses only $\mathop{\mathrm{WMD}}\nolimits(x,y)$; see Equation \ref{['eq:wmd-combine']}. Here, $n=1$.
  • Figure 5: Pearson's r correlations on MLQE-PE for [wrd $\oplus$ snt] when fine-tuning on limited amounts of parallel data. We explore sample sizes of 10k, 20k, 30k, and 200k.